Analysis of Cancer Research Discussion Text and References in High-Impact Factor Journals for Possible Indicators of Paper Mills
Bibliographic record
Abstract
Annie Whamond,1 Adrian G. Barnett,2 Jennifer A. Byrne1,3 Objective Paper mills are unethical organizations that provide low-value or fraudulent content to client authors.1 Although image manipulation and incorrect reagent detection tools are available,1 these approaches can require reviewers and readers to learn new skills. Scaled manuscript production through templates may also result in paper mill articles showing other common features. We aimed to help readers identify possible indicators of paper mill support by simply scanning article text and references. We focused on discussion sections for this study. Design We used an exploratory, cross-sectional design2 to develop a descriptive analysis of discussion text and references in high Impact Factor (IF) cancer journals, defined as IF of 7 or higher for journal categories oncology, biochemistry and molecular biology, or cell biology. We downloaded the Retraction Watch database on October 16, 2024, and filtered for retraction reason paper mill, subject cancer, and journals on our high IF list. To identify comparison article cohorts, we randomly sampled original cancer articles in (1) the same 8 journals as retracted articles or (2) 8 independent journals with no paper mill retractions and high IF maintained for 20 years or longer. For each article, we recorded the percentages of references first cited in the introduction, methods, results, or discussion section. Focusing on discussion sections, we classified each sentence as providing background, summary, comparison, interpretation or implication, limitation, or future direction3 and recorded whether individual discussion sentences cited new references. Results We found 22 retracted paper mill articles from 8 journals published between June 21, 2016, and June 16, 2022. Both comparison groups (50 articles each) were restricted to articles published between January 1, 2016, and October 31, 2024. Articles in all 3 groups included similar numbers of total references per article (median [IQR]: 42 [36-46] retracted articles; 47 [36-55] articles in same journals; 51 [45-60] articles in independent journals) and percentages of discussion sentences per article (median [IQR]: 34 [30-38] retracted articles; 40 [30-48] same journals; 34 [26-43] independent journals). Analyses of discussion sections indicated that retracted paper mill articles included higher percentages of references that were first cited in the discussion and higher percentages of discussion sentences that described background information and cited new references (Figure 25-1022). https://assets.underline.io/markdown_image/1/image/d20ae00ed6b6328fe12ed01f027e066e.png Conclusions Our analyses suggest that some discussion sections in retracted paper mill cancer research articles reiterate background information that is supported by new and possibly superfluous references. While recognizing that genuine studies also cite new references in discussion sections, superficial and redundant second introductions in discussion sections could help readers identify potentially problematic articles in high IF cancer journals, particularly when combined with other features of paper mill support.1 References 1. Byrne JA, Abalkina A, Akinduro-Aje O, et al. A call for research to address the threat of paper mills. PLoS Biol. 2024;22(11):e3002931. doi:10.1371/journal.pbio.3002931 2. Kesmodel US. Cross-sectional studies—what are they good for? Acta Obstet Gynecol Scand. 2018;97(4):388-393. doi:10.1111/aogs.13331 3. Toronto CE, Remington R. Discussion and conclusion. In: A Step-by-Step Guide to Conducting an Integrative Review. Toronto CE, Remington R, eds. Springer International; 2020:71-84. 1School of Medical Sciences, Faculty of Medicine and Health, University of Sydney, Sydney, New South Wales, Australia, jennifer.byrne@health.nsw.gov.au; 2School of Public Health and Social Work, Queensland University of Technology, Kelvin Grove, Queensland, Australia; 3NSW Health Statewide Biobank, NSW Health Pathology, Camperdown, New South Wales, Australia. Conflict of Interest Disclosures None reported. Funding/Support Adrian G. Barnett and Jennifer A. Byrne acknowledge grant funding from the National Health and Medical Research Council of Australia (ideas grant APP2029249). This grant supports Annie Whamond’s PhD candidature. Role of Funder/Sponsor The funding body played no role in the study design, data collection, management, analysis, or interpretation, and will play no role in the writing of any report, or the decision to submit the report for publication.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.052 | 0.285 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.026 | 0.028 |
| Science and technology studies | 0.002 | 0.002 |
| Scholarly communication | 0.005 | 0.004 |
| Open science | 0.002 | 0.002 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.017 | 0.005 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".