Hybrid Distributed Stochastic Gradient Descent for Federated Learning
Bibliographic record
Abstract
With the advancement of information technology in the past decades, the world embraces the era of 'Big Data', in which large volumes of data are being produced in high velocity, while there is an increasing demand in processing these data. Such environment sets up a perfect playground for deep learning, which is able to utilize the large volumes of data to achieve various tasks. However, as both the volumes of data and the complexity of neural network architecture rises, it becomes increasingly expensive to train the model on a single machine. Federated learning becomes a hot research topic in recent years, which decentralizes the conventional deep learning architecture by distributing both data storage and/or computation operations to multiple machines, while it requires no exchange of information about the local training data so that the data privacy is preserved. In the literature, two different transmission approaches for federated learning, analog-based transmissions and digital-based transmissions, were studied and it was shown that the analog-based approach considerably outperforms the digital-based approach by utilizing the waveform superposition principle of the wireless access medium. In this thesis, we propose the Hybrid Distributed Stochastic Gradient Descent (Hybrid DSGD), a training scheme for federated learning which utilizes the advantages of both digital and analog transmissions to reduce communication overhead and latency. We demonstrate why the conventional analog-based transmission schemes perform poorly when the number of workers participating the training and/or the power available for each worker are restricted. We then explain how our scheme addresses such issue. We will show through experiments that the hybrid DSGD is able to outperform the conventional analog-based transmission scheme under such circumstance.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.003 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.000 |
| Bibliometrics | 0.000 | 0.001 |
| Science and technology studies | 0.000 | 0.001 |
| Scholarly communication | 0.001 | 0.001 |
| Open science | 0.002 | 0.001 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.002 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".