Distributed Optimization Algorithms with Improved Efficiency, Reliability, and Privacy Preservation

With machine learning growing in popularity and relying more on big data and large-scale computations, the demand for distributed computing has continuously increased. This thesis is motivated by the challenges that have emerged as a result of this demand. The first challenge is the spread of computations across multiple processing nodes with varying and unpredictable speeds, leading to per-node delays, known as "straggler" issues. The second challenge is the result of data contribution from sources with diverse privacy requirements, making the development of equitable privacy-preserving approaches for data usage more complex. Motivated by these challenges, the following three research themes form the main contributions of this thesis. In the first research theme, we incorporate approximation techniques to develop a more efficient coded distributed computing (CDC) approach for mitigating straggler issues. CDC relies on error correction codes to introduce "coded" redundant tasks that are not pure replicas of the original ones. Progress in CDC has mainly focused on the realization of exact computation recovery once a sufficient number of nodes complete their tasks. However, many computational problems involve randomness and are, therefore, naturally tolerant of inexact results. In the thesis, we develop rate-distortion analogs for CDC and design "approximated" CDC schemes in which there are multiple stages of inexact recovery en route to exact recovery. As we will see in the thesis, our schemes help accelerate computation recovery by balancing between accuracy and speed. In the second and third themes, we incorporate privacy guarantees via differential privacy (DP) to develop new distributed learning methods that are tailored to the varying privacy requirements of participating nodes. Differentially-private federated learning (DP-FL) is an emerging distributed learning framework that enables nodes to collaboratively train a shared model while anonymizing their local data. Progress on DP-FL has mainly relied on implicit assumptions that nodes share the same level of trust with all other participants, have equal privacy constraints, and have fixed incentives to collaborate over time (learning iterations). In the second theme, we extend DP-FL to reflect a setting with multiple, potentially overlapping groups of nodes with varying intra- vs. inter-group levels of trust and analyze how we can control privacy leakage propagation across groups. In the third theme, we show that time variation of the standard DP-FL parameters can improve utility while adhering to personalized privacy constraints. In the thesis, we show our contributions to these two themes, which help incentivize greater participation and enhance participant equity.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.