SP46. Using ChatGPT to Review the Literature: A Cautionary Tale
Bibliographic record
Abstract
PURPOSE: ChatGPT has shown impressive results in the medical field, recently matching residents in the Plastic Surgery In-Service Examinations. In academic writing, ChatGPT can generate ideas, organize thinking, and rewrite difficult sections of a paper, although its proper ethical use is still highly debated. With ChatGPT being a potential tool for scientific writing while being barred from authorship in most peer-reviewed journals, we seek to document the abilities of such technologies and consider their appropriate applications in publication. Herein we define the strengths and weaknesses of ChatGPT in writing a literature review on autologous fat grafting. METHODS: ChatGPT-4o (OpenAI, San Francisco, CA, USA) was used to generate a literature review article from ideation to final editing. ChatGPT was asked for three topics within plastic and reconstructive surgery to review, with autologous fat grafting chosen from the provided ideas. ChatGPT was prompted to create an outline and then write each section with corresponding citations. The references were evaluated for accuracy via a person-supervised PubMed search. Final editing was accomplished by asking ChatGPT to match the tone and style of a published narrative review. The writing was compared to published work through a survey of medical professionals. One paragraph was put in two AI detectors, WinstonAI (Montreal, Quebec, CAN) and ZeroGPT (Casper, WY, USA). RESULTS: ChatGPT brainstormed three topics in plastic and reconstructive surgery — biomaterials in tissue engineering, autologous fat grafting, and scar management. Autologous fat grafting was selected and ChatGPT provided a clear outline with subtopics including the application, techniques, and challenges of fat grafting. After prompting, ChatGPT successfully wrote two paragraphs for each section, resulting in a cohesive overview of autologous fat grafting. It then edited the content to match the tone and style of the published narrative review it was provided, making it difficult to distinguish from human authorship. In a survey of trainees, attendings, and researchers, 53% correctly identified the abstract written by ChatGPT. 67% of respondents indicated they would not suspect AI input if the abstract were in a scientific journal. Further analysis of the AI written content revealed vague statements and erroneous citations. Of the 21 citations, 5 were correct, 8 had errors in the citation, and 8 could not be found in PubMed. When asked to summarize an imagined citation, ChatGPT fabricated a study, complete with methods and results. When provided a real citation, ChatGPT misrepresented the results, adding in additional variables and statistical significance. Once provided with the entire paper, ChatGPT generated an accurate summary. CONCLUSIONS: ChatGPT-4o performed well in suggesting scientific topics, generating an organized outline, and editing provided material. Its writing was professional and difficult to distinguish from human-authored material. However, ChatGPT failed to accurately cite existing sources and fabricated entire studies. By leading ChatGPT through a literature review, we have defined successful use cases in academic writing, as well as areas to approach with caution. As with any tool, authors must adhere to the standards of their targeted journal and take full responsibility for all submitted work.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.150 | 0.520 |
| Meta-epidemiology (narrow) | 0.002 | 0.002 |
| Meta-epidemiology (broad) | 0.002 | 0.002 |
| Bibliometrics | 0.007 | 0.006 |
| Science and technology studies | 0.004 | 0.009 |
| Scholarly communication | 0.011 | 0.012 |
| Open science | 0.004 | 0.010 |
| Research integrity | 0.010 | 0.014 |
| Insufficient payload (model declined to judge) | 0.041 | 0.030 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".