Abstract 208: Development of Evidence Statement curation algorithms to aid cancer variant interpretation
Bibliographic record
Abstract
Abstract The Clinical Interpretation of Variants in Cancer (CIViC) knowledgebase (civicdb.org) is an open access, centralized hub for structured, community curated and expertly moderated relationships between genomic variants and cancer. Evidence is curated from peer-reviewed, published literature and is classified into one of five Types: Predisposing, Diagnostic, Prognostic, Predictive (therapeutic), or Functional. The robustness of the Evidence is conveyed through the assignment of Levels with the first three derived from patient studies (Validated, Clinical, Case Study), Preclinical, generated from in vivo or in vitro data, and Inferential, which describes indirect associations. Each Evidence Item requires an Evidence Statement written in the curator's own words summarizing the source's results regarding the variant's clinical impact. Collaborations with groups like ClinGen have generated a significant influx of new curators, increasing the demand for detailed principles regarding data prioritization in the Evidence Statement in order to streamline the curation process. The curation community would benefit from simpler, visual guides through the complex decisions needed to appropriately and consistently curate Evidence Items. We are devoting significant effort to continue the development of straightforward Evidence curation algorithms (decision trees) similar to those used in clinical molecular testing labs to aid CIViC curators. Previously published guidelines on development of these statements are the basis of our Evidence algorithms. Obvious inflection points for curators are clearly identified with specific details noted for each to optimize decision efficiency. As the predominant Evidence Type comprising 57% of all CIViC submissions, 58% of referenced patient trials, and 92% of Preclinical submissions, Predictive Evidence is the initial focus of our pilot guidelines with Diagnostic and Prognostic to follow. Within the Predictive Evidence Type, clinical trials, case studies, and preclinical Levels each require vastly different Evidence Statement details and ultimately the creation of three separate, uniquely modeled algorithms. The implementation of these algorithms will assist in streamlining both curation and the expert review process. Notably, a template is not being created, as the preservation of curator style and voice is important to maintain the community feel of the database. To ensure the highest level of clarity, our team is utilizing specific novice and experienced curators to assist with the development process. As these algorithms pass the pilot phase, they are being tested as curator training tools. Ultimately, these guidelines will be used to encourage independence in curators and to enhance the Evidence already contained in CIViC. Citation Format: Jason Saliba, Lana Sheta, Kilannin Krysiak, Arpad Danos, Alex Marr, Erica Barnell, Shahil Pema, Wan-Hsin Lin, Panieh Terraf, Joshua F. McMichael, Cameron J. Grisdale, Shruti Rao, Susanna Kiwala, Adam Coffman, Alex Wagner, Obi L. Griffith, Malachi Griffith. Development of Evidence Statement curation algorithms to aid cancer variant interpretation [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2021; 2021 Apr 10-15 and May 17-21. Philadelphia (PA): AACR; Cancer Res 2021;81(13_Suppl):Abstract nr 208.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.072 | 0.344 |
| Meta-epidemiology (narrow) | 0.002 | 0.002 |
| Meta-epidemiology (broad) | 0.002 | 0.005 |
| Bibliometrics | 0.016 | 0.008 |
| Science and technology studies | 0.002 | 0.002 |
| Scholarly communication | 0.012 | 0.006 |
| Open science | 0.005 | 0.007 |
| Research integrity | 0.004 | 0.005 |
| Insufficient payload (model declined to judge) | 0.032 | 0.015 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".