Mixture-of-Experts as Continual Knowledge Adapter for Mobile Vision Understanding
Bibliographic record
Abstract
Continual machine learning in the context of limited computational resources and data availability is critical in the connected digital world. Current intelligent applications predominantly rely on deep learning models requiring labor/computation-intensive training. These models often struggle to adapt effectively to new data while preserving performance on previously learned knowledge. In this paper, we introduce a lightweight method for continual knowledge adaptation that can address these challenges. To prevent disruption of the existing services, we propose a Mixture-of-Experts (MoE) adapter that integrates seamlessly with the existing vision model to encode new data. The weights of the original model are kept fixed during the adaptation process, ensuring the preservation of previously learned knowledge. The MoE technique enables scaling up the parameters of the adapter while maintaining a relatively low computation, making it fit for constrained devices in mobile computation scenarios. Furthermore, to enhance learning efficiency and accelerate convergence with new data, we implement a knowledge fusion mechanism that facilitates interaction between the existing knowledge and the information extracted from new data. The timing of employing the fusion module is further investigated. We find that it is conducive in scenarios where the task's performance requirements are enhanced. The MoE adapter and knowledge fusion module are integrated at each stage with minimal trainable parameters, efficiently optimizing resource usage. Extensive experiments and ablation studies validate the effectiveness of the proposed method. Specifically, the proposed method prevents an accuracy drop of 43.02% on the previous data compared to the continual train method, while achieving an accuracy of 44.81% on the new data, which is even 0.34% higher than fully training a new model.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.003 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.001 | 0.001 |
| Science and technology studies | 0.000 | 0.001 |
| Scholarly communication | 0.001 | 0.003 |
| Open science | 0.002 | 0.003 |
| Research integrity | 0.002 | 0.002 |
| Insufficient payload (model declined to judge) | 0.002 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".