Abstract B007: TheBlueScrubs-v1: A Large-Scale Curated Dataset with ∼11 Billion Oncology Tokens for AI-Driven Cancer Research
Bibliographic record
Abstract
Abstract Large language models (LLMs) are increasingly pivotal in cancer research, yet current public datasets offer insufficient scale and diversity to capture the complexity of oncology. To address this gap, we created TheBlueScrubs-v1, a 25-billion-token corpus of medical texts curated from the SlimPajama dataset. Approximately one-third of these tokens (∼11 billion) are annotated as cancer-related, making this one of the largest public, domain-focused text collections available for training and benchmarking oncology LLMs. Our two-stage pipeline first applied a high-speed logistic regression classifier (trained on a balanced set of 60,000 medical vs. non-medical documents) to label texts by medical relevance. This process extracted ∼4% of SlimPajama, yielding documents with at least 0.8 probability of containing medical content. Next, a 70B-parameter open-source LLM (Llama 3.1) evaluated each text’s medical scope, factual precision, and safety on 1–5 scales. Validation by clinicians and GPT-4o found strong concordance, confirming the reliability of these automated assessments. We further developed a specialized cancer classifier using logistic regression with TF-IDF features, trained on 60,000 examples, to identify oncology-related texts. This yielded a high-quality oncology subset (∼11 billion tokens) spanning topics such as cancer diagnosis, therapeutics, and real-world clinical notes. Detailed safety metrics enable red-teaming to mitigate misinformation and promote ethical use in oncology research. Potential applications include (1) fine-tuning LLMs for oncology-focused tasks such as treatment recommendation, clinical trial matching, and patient education, (2) building safety classifiers to detect harmful or misleading content, and (3) synthetic data generation to expand training sets while preserving privacy. Early experiments demonstrate that LLMs fine-tuned on TheBlueScrubs-v1 achieve performance on par with or exceeding models trained on smaller, specialized medical corpora. By releasing this large-scale, annotated dataset under an open license, we aim to accelerate innovation in AI-driven cancer research and foster collaborative efforts toward safer, more accurate clinical language models. Citation Format: Luis Felipe, Gilmer Valdes. TheBlueScrubs-v1: A Large-Scale Curated Dataset with ∼11 Billion Oncology Tokens for AI-Driven Cancer Research [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Artificial Intelligence and Machine Learning; 2025 Jul 10-12; Montreal, QC, Canada. Philadelphia (PA): AACR; Clin Cancer Res 2025;31(13_Suppl):Abstract nr B007.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.003 | 0.014 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.002 |
| Bibliometrics | 0.004 | 0.004 |
| Science and technology studies | 0.002 | 0.001 |
| Scholarly communication | 0.002 | 0.002 |
| Open science | 0.003 | 0.003 |
| Research integrity | 0.003 | 0.002 |
| Insufficient payload (model declined to judge) | 0.015 | 0.023 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".