Improving deep neural networks for LVCSR using rectified linear units and dropout
Why is this work in the frame?
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Machine scores (provisional)
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
- Teacher spread
- 0.207 · how far apart the two teachers sit on this one work
- Validation status
score_only:v0-immature-baseline· verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it
Abstract
Recently, pre-trained deep neural networks (DNNs) have outperformed traditional acoustic models based on Gaussian mixture models (GMMs) on a variety of large vocabulary speech recognition benchmarks. Deep neural nets have also achieved excellent results on various computer vision tasks using a random “dropout” procedure that drastically improves generalization error by randomly omitting a fraction of the hidden units in all layers. Since dropout helps avoid over-fitting, it has also been successful on a small-scale phone recognition task using larger neural nets. However, training deep neural net acoustic models for large vocabulary speech recognition takes a very long time and dropout is likely to only increase training time. Neural networks with rectified linear unit (ReLU) non-linearities have been highly successful for computer vision tasks and proved faster to train than standard sigmoid units, sometimes also improving discriminative performance. In this work, we show on a 50-hour English Broadcast News task that modified deep neural networks using ReLUs trained with dropout during frame level training provide an 4.2% relative improvement over a DNN trained with sigmoid units, and a 14.4% relative improvement over a strong GMM/HMM system. We were able to obtain our results with minimal human hyper-parameter tuning using publicly available Bayesian optimization code.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
The record
- Venue
- Topic
- Speech Recognition and Synthesis
- Field
- Computer Science
- Canadian institutions
- University of Toronto
- Funders
- —
- Keywords
- Computer scienceDropout (neural networks)Artificial neural networkSpeech recognitionArtificial intelligenceSigmoid functionDiscriminative modelDeep neural networksTask (project management)VocabularyDeep learningMixture modelHidden Markov modelTime delay neural networkBayesian probabilityPattern recognition (psychology)Machine learning
- Has abstract in OpenAlex
- yes