On the importance of initialization and momentum in deep learning
Why is this work in the frame?
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Abstract
Deep and recurrent neural networks (DNNs and RNNs respectively) are powerful models that were considered to be almost impossible to train using stochastic gradient descent with momentum. In this paper, we show that when stochastic gradient descent with momentum uses a well-designed random initialization and a particular type of slowly increasing schedule for the momentum parameter, it can train both DNNs and RNNs (on datasets with long-term dependencies) to levels of performance that were previously achievable only with Hessian-Free optimization. We find that both the initialization and the momentum are crucial since poorly initialized networks cannot be trained with momentum and well-initialized networks perform markedly worse when the momentum is absent or poorly tuned. Our success training these models suggests that previous attempts to train deep and recurrent neural networks from random initializations have likely failed due to poor initialization schemes. Furthermore, carefully tuned momentum methods suffice for dealing with the curvature issues in deep and recurrent network training objectives without the need for sophisticated second-order methods. 1.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
The record
- Venue
- Topic
- Stochastic Gradient Optimization Techniques
- Field
- Computer Science
- Canadian institutions
- University of Toronto
- Funders
- —
- Keywords
- InitializationMomentum (technical analysis)Computer scienceRecurrent neural networkGradient descentStochastic gradient descentDeep learningArtificial intelligenceDeep neural networksArtificial neural networkScheduleMachine learningMathematical optimizationAlgorithmMathematics
- Has abstract in OpenAlex
- yes