Not a walk in the park: the ECVAM whole embryo culture model challenged with pharmaceuticals and attempted improvements with random forest design
Bibliographic record
Abstract
BACKGROUND: The European Committee for the Validation of Alternative Methods (ECVAM) supported the development of a linear discriminant embryotoxicity prediction model founded on rat whole embryo culture (Piersma et al. (2004). Altern Lab Anim 32:275–307). Our goals were to (1) assess the accuracy of this model with pharmaceuticals, and (2) to use the data to develop a more accurate prediction model. METHODS: Sixty-one chemicals of known in vivo activity were tested. They were part of the ECVAM validation set (N513), commercially available pharmaceuticals (N531), and Pfizer chemicals that did not reach the market, but for which developmental toxicity data were available (N517). They were tested according to the ECVAM procedures. Fifty-seven of these chemicals were used for Random Forest modeling to develop an alternate model with the goal of using surrogate endpoints for simplified assessments and to improve the predictivity of the model. RESULTS: Using part of the ECVAM chemical test set, the ECVAM prediction model was 77% accurate. This approximated what was reported in the validation study (80%; Piersma et al. (2004). Altern Lab Anim 32:275–307). However, when confronted with novel chemicals, the accuracy of the linear discriminant model dropped to 56%. In an attempt to improve this performance, we used a Random Forest model that provided rankings and confidence estimates. Although the model used simpler endpoints, its performance was no better than the ECVAM linear discriminant model. CONCLUSIONS: This study confirms previous concerns about the applicability of the ECVAM prediction model to a more diverse chemical set, and underscores the challenges associated with developing embryotoxicity prediction models.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.012 | 0.009 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.002 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.001 |
| Scholarly communication | 0.001 | 0.001 |
| Open science | 0.001 | 0.001 |
| Research integrity | 0.001 | 0.002 |
| Insufficient payload (model declined to judge) | 0.001 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".