SP46. Using ChatGPT to Review the Literature: A Cautionary Tale
Notice bibliographique
Résumé
PURPOSE: ChatGPT has shown impressive results in the medical field, recently matching residents in the Plastic Surgery In-Service Examinations. In academic writing, ChatGPT can generate ideas, organize thinking, and rewrite difficult sections of a paper, although its proper ethical use is still highly debated. With ChatGPT being a potential tool for scientific writing while being barred from authorship in most peer-reviewed journals, we seek to document the abilities of such technologies and consider their appropriate applications in publication. Herein we define the strengths and weaknesses of ChatGPT in writing a literature review on autologous fat grafting. METHODS: ChatGPT-4o (OpenAI, San Francisco, CA, USA) was used to generate a literature review article from ideation to final editing. ChatGPT was asked for three topics within plastic and reconstructive surgery to review, with autologous fat grafting chosen from the provided ideas. ChatGPT was prompted to create an outline and then write each section with corresponding citations. The references were evaluated for accuracy via a person-supervised PubMed search. Final editing was accomplished by asking ChatGPT to match the tone and style of a published narrative review. The writing was compared to published work through a survey of medical professionals. One paragraph was put in two AI detectors, WinstonAI (Montreal, Quebec, CAN) and ZeroGPT (Casper, WY, USA). RESULTS: ChatGPT brainstormed three topics in plastic and reconstructive surgery — biomaterials in tissue engineering, autologous fat grafting, and scar management. Autologous fat grafting was selected and ChatGPT provided a clear outline with subtopics including the application, techniques, and challenges of fat grafting. After prompting, ChatGPT successfully wrote two paragraphs for each section, resulting in a cohesive overview of autologous fat grafting. It then edited the content to match the tone and style of the published narrative review it was provided, making it difficult to distinguish from human authorship. In a survey of trainees, attendings, and researchers, 53% correctly identified the abstract written by ChatGPT. 67% of respondents indicated they would not suspect AI input if the abstract were in a scientific journal. Further analysis of the AI written content revealed vague statements and erroneous citations. Of the 21 citations, 5 were correct, 8 had errors in the citation, and 8 could not be found in PubMed. When asked to summarize an imagined citation, ChatGPT fabricated a study, complete with methods and results. When provided a real citation, ChatGPT misrepresented the results, adding in additional variables and statistical significance. Once provided with the entire paper, ChatGPT generated an accurate summary. CONCLUSIONS: ChatGPT-4o performed well in suggesting scientific topics, generating an organized outline, and editing provided material. Its writing was professional and difficult to distinguish from human-authored material. However, ChatGPT failed to accurately cite existing sources and fabricated entire studies. By leading ChatGPT through a literature review, we have defined successful use cases in academic writing, as well as areas to approach with caution. As with any tool, authors must adhere to the standards of their targeted journal and take full responsibility for all submitted work.
Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.
Comment cette classification a été obtenuedéplier
Prédiction distillée sur la base complète
Imitation des enseignantsNi prévalence calibrée, ni vérité terrain. Validation humaine à venir. Apprise à partir de 10 348 étiquettes directes de Codex et de 10 348 étiquettes directes de Gemma. Le mode candidate est l'union des têtes enseignantes seuillées; le consensus est leur intersection. Ces sorties portent le statut machine_predicted_unvalidated et ne sont ni des étiquettes humaines ni des étiquettes directes de modèles de pointe.
Scores Codex et Gemma par catégorie
| Catégorie | Codex | Gemma |
|---|---|---|
| Métarecherche | 0,001 | 0,003 |
| Méta-épidémiologie (sens strict) | 0,000 | 0,000 |
| Méta-épidémiologie (sens large) | 0,000 | 0,000 |
| Bibliométrie | 0,000 | 0,001 |
| Études des sciences et des technologies | 0,000 | 0,000 |
| Communication savante | 0,000 | 0,000 |
| Science ouverte | 0,000 | 0,000 |
| Intégrité de la recherche | 0,000 | 0,000 |
| Charge utile insuffisante (le modèle a refusé de juger) | 0,000 | 0,000 |
Scores machine (provisoires)
Les deux têtes enseignantes du modèle étudiant, lues sur ce travail. Un score ordonne la base pour la relecture; il n'affirme jamais une catégorie, et le statut de validation accompagne chaque rangée tel quel.
Scores de référence d'un modèle non mature (critères de maturité non atteints, 7 itérations). Un score ordonne; il n'affirme jamais une catégorie.
score_only:v0-immature-baseline · tel quel depuis la passe de notation : score_only signifie que le nombre peut ordonner les travaux, et qu'aucune étiquette de catégorie n'en découleClassification
machine, non validéePrédiction automatique; un appel candidat d’une seule tête enseignante, pas un consensus.
Le détail, modèle par modèle et score par score, se trouve en fin de page sous « Comment cette classification a été obtenue ».