Bibliographic record
Abstract
Knowledge distillation is a powerful and flexible machine learning technique that can be used to train smaller, more efficient models (called students) by mimicking larger trained models (called teachers). Such students can often achieve better predictive performance than models trained in a classical supervised manner. However, despite its empirical success, a rigorous foundation of knowledge distillation is still largely non-existent. This dissertation investigates both the theoretical and empirical foundations of knowledge distillation. In the first two papers, we develop theoretical frameworks for understanding knowledge distillation in the simplified settings of self-distillation with kernel ridge regression and Gaussian process models, respectively. In these frameworks, we investigate the properties of iterative self-distillation and determine particular regularizing behaviors imposed by self-distillation. In a third paper, we perform a rigorous empirical study of knowledge distillation with neural networks to support our theoretical findings. We investigate the efficiency of knowledge distillation under various controlled settings to determine under which conditions we can obtain perfect teacher-student agreement. In a fourth paper, we illustrate the real-world applicability of knowledge distillation by showing how to apply knowledge distillation for personalized automatic sleep scoring based on Ear-EEG measurements. Finally, in a fifth paper, we address the challenge of exploiting diverse publicly available neural network models to improve the predictive performance on a given task under computational constraints. In particular, we propose a method to construct efficient models by identifying and distilling suitable pre-trained models with minimal access to these models. Overall, this dissertation contributes to the theoretical and empirical foundations of knowledge distillation and proposes novel methods for adapting publicly available neural network models to specific tasks under constraints.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.005 | 0.019 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.002 |
| Bibliometrics | 0.001 | 0.002 |
| Science and technology studies | 0.001 | 0.003 |
| Scholarly communication | 0.002 | 0.005 |
| Open science | 0.003 | 0.004 |
| Research integrity | 0.002 | 0.007 |
| Insufficient payload (model declined to judge) | 0.004 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".