Highland games: A benchmarking exercise in predicting biophysical and drug properties of monoclonal antibodies from amino acid sequences
Bibliographic record
Abstract
Biopharmaceutical product and process development do not yet take advantage of predictive computational modeling to nearly the degree seen in industries based on smaller molecules. To assess and advance progress in this area, spirited coopetition (mutually beneficial collaboration between competitors) was successfully used to motivate industrial scientists to develop, share, and compare data and methods which would normally have remained confidential. The first "Highland Games" competition was held in conjunction with the October 2018 Recovery of Biological Products Conference in Ashville, NC, with the goal of benchmarking and assessment of the ability to predict development-related properties of six antibodies from their amino acid sequences alone. Predictions included purification-influencing properties such as isoelectric point and protein A elution pH, and biophysical properties such as stability and viscosity at very high concentrations. Essential contributions were made by a large variety of individuals, including companies which consented to provide antibody amino acid sequences and test materials, volunteers who undertook the preparation and experimental characterization of these materials, and prediction teams who attempted to predict antibody properties from sequence alone. Best practices were identified and shared, and areas in which the community excels at making predictions were identified, as well as areas presenting opportunities for considerable improvement. Predictions of isoelectric point and protein A elution pH were especially good with all-prediction average errors of 0.2 and 1.6 pH unit, respectively, while predictions of some other properties were notably less good. This manuscript presents the events, methods, and results of the competition, and can serve as a tutorial and as a reference for in-house benchmarking by others. Organizations vary in their policies concerning disclosure of methods, but most managements were very cooperative with the Highland Games exercise, and considerable insight into common and best practices is available from the contributed methods. The accumulated data set will serve as a benchmarking tool for further development of in silico prediction tools.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.016 | 0.021 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.002 | 0.001 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.003 | 0.002 |
| Open science | 0.003 | 0.003 |
| Research integrity | 0.002 | 0.002 |
| Insufficient payload (model declined to judge) | 0.005 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".