Wasserstein-Based Feature Map Knowledge Transfer to Improve the Performance of Small Deep Neural Networks
Bibliographic record
Abstract
The Convolution Neural Network (CNN) is a class of Deep Neural Networks, specifically used for computer vision applications. The CNN architecture consists of several hidden layers. This hidden layer information, which is referred to as a feature map, contains rich spatial and semantic information about the input data. A few techniques like Knowledge Distillation, Deep Mutual Learning, and Adversarial Deep Mutual Learning have been introduced to transfer the knowledge of a large DNN to a smaller DNN for improving its prediction accuracy. However, the existing methods do not effectively use feature maps as a source of information that can be exploited to improve the performance of the small DNN. We propose an adversarial learning-based approach that consists of a simple generator and discriminator network for transferring feature map information from a large pre-trained DNN to a smaller DNN using the Wasserstein metric. Our approach helps the smaller DNNs generate feature maps similar to a large pre-trained DNN, thereby improving the accuracy and generalization ability. Our experiments show that a variety of small DNN networks benefit from the proposed approach and achieve compelling results on the CIFAR-100 and CIFAR-10 benchmark datasets. Moreover, experimental results on the MobileNet architecture illustrate that the proposed approach is particularly effective for relatively small DNN networks.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.000 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.001 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.002 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".