Why Does Unsupervised Pre-training Help Deep Learning?
Why is this work in the frame?
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Machine scores (provisional)
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
- Teacher spread
- 0.230 · how far apart the two teachers sit on this one work
- Validation status
score_only:v0-immature-baseline· verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it
Abstract
Much recent research has been devoted to learning algorithms for deep architectures such as Deep Belief Networks and stacks of autoencoder variants with impressive results being obtained in several areas, mostly on vision and language datasets. The best results obtained on supervised learning tasks often involve an unsupervised learning component, usually in an unsupervised pre-training phase. The main question investigated here is the following: why does unsupervised pre-training work so well? Through extensive experimentation, we explore several possible explanations discussed in the literature including its action as a regularizer (Erhan et al., 2009b) and as an aid to optimization (Bengio et al., 2007). Our results build on the work of Erhan et al. (2009b), showing that unsupervised pre-training appears to play predominantly a regularization role in subsequent supervised training. However our results in an online setting, with a virtually unlimited data stream, point to a somewhat more nuanced interpretation of the roles of optimization and regularization in the unsupervised pre-training effect.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
The record
- Venue
- Topic
- Domain Adaptation and Few-Shot Learning
- Field
- Computer Science
- Canadian institutions
- Université de Montréal
- Funders
- —
- Keywords
- Artificial intelligenceUnsupervised learningComputer scienceMachine learningDeep learningRegularization (linguistics)GeneralizationAutoencoderDeep belief networkSemi-supervised learningCompetitive learningTraining (meteorology)Mathematics
- Has abstract in OpenAlex
- yes