Random search for hyper-parameter optimization
Why is this work in the frame?
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Abstract
Grid search and manual search are the most widely used strategies for hyper-parameter optimization. This paper shows empirically and theoretically that randomly chosen trials are more efficient for hyper-parameter optimization than trials on a grid. Empirical evidence comes from a comparison with a large previous study that used grid search and manual search to configure neural networks and deep belief networks. Compared with neural networks configured by a pure grid search, we find that random search over the same domain is able to find models that are as good or better within a small fraction of the computation time. Granting random search the same computational budget, random search finds better models by effectively searching a larger, less promising configuration space. Compared with deep belief networks configured by a thoughtful combination of manual search and grid search, purely random search over the same 32-dimensional configuration space found statistically equal performance on four of seven data sets, and superior performance on one of seven. A Gaussian process analysis of the function from hyper-parameters to validation set performance reveals that for most data sets only a few of the hyper-parameters really matter, but that different hyper-parameters are important on different data sets. This phenomenon makes
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
The record
- Venue
- Topic
- Advanced Multi-Objective Optimization Algorithms
- Field
- Computer Science
- Canadian institutions
- Université de Montréal
- Funders
- —
- Keywords
- Hyperparameter optimizationRandom searchComputer scienceGridSet (abstract data type)Artificial neural networkFraction (chemistry)Search algorithmData miningArtificial intelligenceMachine learningAlgorithmMathematics
- Has abstract in OpenAlex
- yes