HarMoEny: Efficient multi-GPU inference of Mixture of Experts models by scheduling experts across accelerators
Bibliographic record
Abstract
Mixture-of-Experts (MoE) models enhance computational efficiency during inference by selectively activating specialized experts for each request.This approach facilitates efficient model scaling on multi-GPU systems leveraging expert parallelism, without compromising performance.However, uneven load distribution across experts often creates imbalances across GPUs, increasing latency due to idle waiting times.To address this, we propose HarMoEny, a novel solution that employs two simple techniques: (i) dynamic token redistribution to underutilized GPUs and (ii) asynchronous prefetching of experts from system to GPU memory.These methods achieve near-perfect load balancing across GPUs and amortize the cost of executing experts not resident in GPU memory, thereby reducing waiting times and latency.We implement HarMoEny and evaluate its performance against four state-of-the-art baselines using real-world and artificially imbalanced datasets.Under heavy load imbalance, HarMoEny improves throughput by 66.7%-68.6%and reduces time to first token by 31.8%-37.2%,compared to the next-best baseline.Furthermore, our ablation study demonstrates that HarMoEny's scheduling policy reduces the GPU idle time by up to 99% relative to the baseline policies.i AbrgLes modles Mixture-of-Experts (MoE) amliorent l'efficacit computationnelle lors de l'infrence en activant slectivement des experts spcialiss pour chaque requte.Cette approche permet une monte en chelle efficace des modles sur des systmes multi-GPU grce au paralllisme d'experts, sans compromettre les performances.Cependant, une rpartition ingale de la charge entre les experts engendre souvent des dsquilibres entre les GPU, augmentant ainsi la latence cause des temps d'attente inactifs.Pour remdier cela, nous proposons HarMoEny, une solution novatrice qui repose sur deux techniques simples : (i) la redistribution dynamique des jetons vers les GPU sous-utiliss et (ii) le prchargement asynchrone des experts de la mmoire systme vers la mmoire GPU.Ces mthodes permettent un quilibrage de charge quasi-parfait entre les GPU et amortissent le cot d'excution des experts non prsents en mmoire GPU, rduisant ainsi les temps d'attente et la latence.Nous avons implment HarMoEny et valu ses performances face quatre rfrences de pointe, en utilisant des jeux de donnes rels et artificiellement dsquilibrs.En cas de fort dsquilibre de charge, HarMoEny amliore le dbit de 66,7 % 68,6 % et rduit le temps jusqu'au premier jeton de 31,8 % 37,2 %, par rapport la meilleure rfrence.De plus, notre tude d'ablation montre que la politique d'ordonnancement de HarMoEny rduit le temps d'inactivit des GPU jusqu' 99 % par rapport aux politiques de rfrence.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.001 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.002 | 0.001 |
| Bibliometrics | 0.001 | 0.001 |
| Science and technology studies | 0.001 | 0.000 |
| Scholarly communication | 0.000 | 0.001 |
| Open science | 0.003 | 0.001 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".