Harnessing the Power of Local Supervision in Federated Learning
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
Federated learning is widely accepted as a privacy-preserving paradigm for training a shared global model across multiple client devices in a collaborative fashion. However, in practice, the significantly limited computational power on client devices has been a major barrier when we wish to train large models with potentially hundreds of millions of parameters. In this paper, we propose a new architecture, referred to as <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Infocomm</small> , that incorporates locally supervised learning in federated learning. With locally supervised learning, the disadvantages of split learning can be avoided by using a more flexible way to offload training from resource constrained clients to a more capable server. <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Infocomm</small> enables parallel training of different modules of the neural network in both the server and clients in a gradient-isolated fashion. The efficacy in reducing both training time and communication time is supported by our theoretical analysis and empirical results. In the scenario involving larger models and fewer available local data, <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Infocomm</small> has been observed to reduce the elapsed time per round by over 37% without sacrificing accuracy compared to both conventional federated learning or directly combining federated learning and split learning, which showcases the advantages of <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Infocomm</small> under power-constrained IoT scenarios.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.002 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.001 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it