The Silent Author in Urology: Quantifying Large Language Model Influence at the Corpus Level
Bibliographic record
Abstract
Introduction The recent increasing use of large language models (LLMs) such as ChatGPT in scientific writing raises concerns about authenticity and accuracy in biomedical literature. This study aims to quantify the prevalence of Artificial Intelligence (AI)-generated text in urology abstracts from 2010 to 2024 and assess its variation across journal categories and impact factor quartiles. Methods A retrospective analysis was conducted on 64,444 unique abstracts from 38 journals in the field of urology published between 2010-2024 and retrieved via the Entrez API from PubMed. Abstracts were then categorized by subspecialty and stratified by impact factor (IF). A synthetic reference corpus of 10,000 abstracts was generated using GPT-3.5-turbo. A mixture model estimated the proportion of AI-generated text annually, using maximum likelihood estimation with Laplace smoothing. Calibration and specificity of the estimator were assessed using pre-LLM abstracts and controlled mixtures of real and synthetic text. Statistical analyses were performed using Python 3.13. Results The proportion of AI-like text was negligible from 2010 to 2019, rising to 1.8% in 2020 and 5.3% in 2024. In 2024, Men’s Health journals showed the highest AI-like text, while Oncology journals had the lowest. Journals in the highest and lowest IF quartiles showed a higher proportion of AI-like text than mid-quartiles. Type-Token Ratio (TTR) remained stable across the study period. Our validation calibration set showed an area under the curve of 0.5326. Conclusion Textual patterns similar to AI-generated language has risen sharply in urology abstracts since ChatGPT’s release in late 2022, and this trend varies by journal type and IF. Patient summary We looked at whether large language models (LLM) such as ChatGPT are influencing the way urology research is written. We found that signs of LLM-like text were rare in abstracts before 2020 but increased in recent years, especially after the release of ChatGPT. Some journal types use these tools more than others. These findings help raise awareness about the growing role of artificial intelligence in scientific communication.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.027 | 0.163 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.006 | 0.005 |
| Science and technology studies | 0.001 | 0.002 |
| Scholarly communication | 0.004 | 0.003 |
| Open science | 0.001 | 0.002 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.004 | 0.002 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".