tag:blogger.com,1999:blog-19803222.post673908449393473707..comments2018-06-14T06:53:24.124-06:00Comments on natural language processing blog: Hyperparameter search, Bayesian optimization and related topicshalhttp://www.blogger.com/profile/02162908373916390369noreply@blogger.comBlogger5125tag:blogger.com,1999:blog-19803222.post-13245198223759129372014-10-17T09:40:51.160-06:002014-10-17T09:40:51.160-06:00See also http://jaberg.github.io/hyperopt/See also http://jaberg.github.io/hyperopt/Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-19803222.post-19953083343275400492014-10-13T15:16:55.796-06:002014-10-13T15:16:55.796-06:00Why not just fit the hyperparameters and parameter...Why not just fit the hyperparameters and parameters together in a hierarchical model? Basically, meta-analysis and what machine learning people call "adaptation" can both be cast as hierarchical modeling techniques. An example is Hal's "embarassingly simple" approach to adaptation, which Finkel and Manning later formulated as a standard hierarchical model.<br /><br />The usual reason not to just fit a hierarchical model is twofold. The first problem is computation. MCMC, which will let you do the exact comptuation you want, namely take a posterior mean point estimate which minimizes expected error, will be too slow for large data sets. Optimization simply fails in hierarchical setting because the density grows without bound as hierarchical variance shrinks to zero (or as David McKay aptly put it in his book, "EM goes boom"). <br /><br />What you can do is approximate with point estimates, also known as empirical Bayes (in a huge misnomer --- it's no more empirical than full Bayes in a hierarchical model). The standard approach is maximum marginal likelihood (MML), as in the lme4 package in R. What you do is marginalize out the low level parameters, optimize the hyperparameters, then fix the hyperparameters to estimate the low-level parameters in a second pass. <br /><br />We're working on doing this in full generality in Stan using Laplace approximations for the marginalization, but we're not quite all the way there yet. We're also working on black-box EP and VB approaches, which may provide better point estimates and better approximations to uncertainty than a simple Laplace estimate based on curvature at the mode.<br /><br />There's a whole book by Brad Efron, the inventor of the bootstrap, on these kinds of techniques with applications to large-scale genomics problems: <i>Large-Scale Inference<br />Empirical Bayes Methods for Estimation, Testing, and Prediction</i>.Bob Carpenterhttp://mc-stan.org/noreply@blogger.comtag:blogger.com,1999:blog-19803222.post-90784863888936055092014-10-11T08:10:48.577-06:002014-10-11T08:10:48.577-06:00Thanks for discussing our work, Hal! You are insi...Thanks for discussing our work, Hal! You are insightful as always. I thought I might just point out a few things in response to your comments. (Please forgive my gratuitous plugging of my own papers.)<br /><br />The idea of learning across multiple data sets is something we find to be very useful, and Kevin Swersky, Jasper Snoek, and I had a <a href="http://people.seas.harvard.edu/~jsnoek/nips2013transfer.pdf" rel="nofollow">paper on it at NIPS</a> last year. Andreas Krause also had <a href="http://las.ethz.ch/files/krause11contextual.pdf" rel="nofollow">a 2011 paper</a> that discusses this setting.<br /><br />The parallelism thing is also something we take seriously and have some good results on in Section 3.3 of <a href="http://hips.seas.harvard.edu/files/snoek-bayesopt-nips-2012.pdf" rel="nofollow">our 2012 NIPS paper</a>. Again, Andreas Krause and colleagues have also contributed to this with their <a href="http://icml.cc/2012/papers/602.pdf" rel="nofollow">2012 ICML paper</a>.<br /><br />Regarding the idea of leveraging problem structure to speed things up, you might like our <a href="http://arxiv.org/pdf/1406.3896.pdf" rel="nofollow">Freeze-Thaw Bayesian Optimization paper</a> on arxiv. The idea is to guess what the final result will be of the inner-loop optimization and build that into the exploration/exploitation strategy without actually finishing jobs about which you are pessimistic.<br /><br />For what it's worth, you might also worry about complicated constraints in your optimization that you can't identify a priori, e.g., "for some complicated hyperparameter regimes, my code just returns NaNs." For this, you might consider <a href="http://auai.org/uai2014/proceedings/individuals/107.pdf" rel="nofollow">Bayesian Optimization with Unknown Constraints</a> by my student Michael Gelbart, or <a href="http://jmlr.org/proceedings/papers/v32/gardner14.pdf" rel="nofollow">this paper by John Cunningham and colleagues</a>.<br /><br />In a final gratuitous plug, the most current versions of Spearmint can be found <a href="https://github.com/HIPS/Spearmint" rel="nofollow">here</a>, and because Spearmint requires some overhead to set up and run, we've also been building a system for "Bayesian optimization as a web service" that you can sign up for at <a href="https://www.whetlab.com/" rel="nofollow">whetlab.com</a>.<br /><br />Ryan AdamsRPAhttps://www.blogger.com/profile/04521971393725683012noreply@blogger.comtag:blogger.com,1999:blog-19803222.post-32120111368635476402014-10-11T06:56:03.628-06:002014-10-11T06:56:03.628-06:00For taking advantage of some of the structure of l...For taking advantage of some of the structure of learning algorithms - for example the iterative nature - look no further than Freeze-Thaw Bayesian Optimization (K Swersky, J Snoek, RP Adams) [1] from June or so. They describe what you suggest here: Looking at the performance early, and only continuing promising candidates.<br /><br />F.<br /><br />[1] http://arxiv.org/pdf/1406.3896Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-19803222.post-50060716192890125822014-10-10T16:21:24.640-06:002014-10-10T16:21:24.640-06:00On using the idea of meta-learning, there has been...On using the idea of meta-learning, there has been some recent work. There was actually a paper from last year's NIPS where they used *multitask* GPs:<br />Multi-Task Bayesian Optimization: http://machinelearning.wustl.edu/mlpapers/paper_files/NIPS2013_5086.pdf)<br /><br />There was also another paper at this year's AISTATS: <br />Efficient Transfer Learning Method for Automatic Hyperparameter Tuning: http://www.cs.cmu.edu/~dyogatam/papers/yogatama+mann.aistats2014.pdf<br /><br />.. and another paper that uses a somewhat simpler idea of using the results from previous runs (on other related data sets) for better initialization on a new data set: Using Meta-Learning to Initialize Bayesian Optimization of Hyperparameters: http://ceur-ws.org/Vol-1201/MetaSel2014-complete.pdf#page=8Piyush Raihttps://www.blogger.com/profile/01083324281027460256noreply@blogger.com