Distributed Optimization Algorithms with Improved Efficiency, Reliability, and Privacy Preservation
Bibliographic record
Abstract
With machine learning growing in popularity and relying more on big data and large-scale computations, the demand for distributed computing has continuously increased. This thesis is motivated by the challenges that have emerged as a result of this demand. The first challenge is the spread of computations across multiple processing nodes with varying and unpredictable speeds, leading to per-node delays, known as "straggler" issues. The second challenge is the result of data contribution from sources with diverse privacy requirements, making the development of equitable privacy-preserving approaches for data usage more complex. Motivated by these challenges, the following three research themes form the main contributions of this thesis. In the first research theme, we incorporate approximation techniques to develop a more efficient coded distributed computing (CDC) approach for mitigating straggler issues. CDC relies on error correction codes to introduce "coded" redundant tasks that are not pure replicas of the original ones. Progress in CDC has mainly focused on the realization of exact computation recovery once a sufficient number of nodes complete their tasks. However, many computational problems involve randomness and are, therefore, naturally tolerant of inexact results. In the thesis, we develop rate-distortion analogs for CDC and design "approximated" CDC schemes in which there are multiple stages of inexact recovery en route to exact recovery. As we will see in the thesis, our schemes help accelerate computation recovery by balancing between accuracy and speed. In the second and third themes, we incorporate privacy guarantees via differential privacy (DP) to develop new distributed learning methods that are tailored to the varying privacy requirements of participating nodes. Differentially-private federated learning (DP-FL) is an emerging distributed learning framework that enables nodes to collaboratively train a shared model while anonymizing their local data. Progress on DP-FL has mainly relied on implicit assumptions that nodes share the same level of trust with all other participants, have equal privacy constraints, and have fixed incentives to collaborate over time (learning iterations). In the second theme, we extend DP-FL to reflect a setting with multiple, potentially overlapping groups of nodes with varying intra- vs. inter-group levels of trust and analyze how we can control privacy leakage propagation across groups. In the third theme, we show that time variation of the standard DP-FL parameters can improve utility while adhering to personalized privacy constraints. In the thesis, we show our contributions to these two themes, which help incentivize greater participation and enhance participant equity.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.003 | 0.014 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.001 | 0.002 |
| Science and technology studies | 0.001 | 0.002 |
| Scholarly communication | 0.002 | 0.003 |
| Open science | 0.002 | 0.003 |
| Research integrity | 0.001 | 0.003 |
| Insufficient payload (model declined to judge) | 0.002 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".