Using GPT-4 to Automate the Generation of Lay Summaries for Cancer Publications
Notice bibliographique
Résumé
Abstract Background Cancer research literature is often riddled with technical jargon that is not digestible to the average person. Individuals interested in research studies may want to contribute through patient partner engagement or sample donation but find the relevant literature overwhelming. Through the generation of lay summaries, previously inaccessible research papers become easier to comprehend, especially for patient partners or data donors. With large language models (LLMs) continuing to advance, so does their capability to summarize large texts. Objectives In this study, we examined whether LLMs can produce lay summaries of scientific literature at-scale, while maintaining readability and accuracy to their source texts. Methods We developed a tool to generate lay summaries of open-access article abstracts and their full texts with GPT-4-Turbo. Prompt development aimed for a target 8th grade reading level assessed with Flesch-Kincaid Grade Level. Human-review metrics were used to evaluate readability and accuracy when generated using abstracts versus full text articles. Results The average Flesch-Kincaid Grade Level Score was 7.13 for abstract-based summaries and 7.39 for full text-based summaries, indicating summaries at around 7th grade reading level. Human-review metrics showed these summaries were of similar readability and accuracy when generated using abstracts versus full text articles, with mean accuracy scores from human review of 7.09 vs 7.42 out of 10 respectively. Additionally, qualitative patient-based assessment indicated these summaries would encourage participation in research studies. Conclusion By generating lay summaries for complex and lengthy research papers, their scientific information becomes accessible to a larger audience, including patient partners interested in contributing to cancer research. Summaries that are easy to understand will allow participants to make informed decisions about their involvement and appreciate the impact of their contributions if and when their results are published. Lay Summary This study explores if artificial intelligence (AI) can help make hard to read cancer research papers easier to understand for members of the public. Problem When people donate cancer tissue samples or participate in research studies, they often want to know how their contributions are being used. However, scientific papers are full of technical language that’s hard for most people to grasp. People in past studies have said this can make them less willing to take part in research. Methods The study created a computer program using AI (GPT-4-Turbo) to turn complex kidney cancer research papers into simple summaries. They tested whether the AI could summarize both short abstracts and full-length papers effectively. They aimed for summaries at a 6th to 8th-grade reading level. This was to follow Canadian and U.S. health communication guidelines. Results The AI created 106 summaries. Computer measures showed the summaries were close to a 7th-grade reading level. Though, researchers had to tell the AI to write for a 2nd-grade audience to achieve this. Of note, summaries from short abstracts were just as accurate and readable as those from full papers. Eighteen volunteers, including five patient partners, reviewed the summaries and rated them for clarity and accuracy. They were rated at around 7 out of 10 points. All patient partners said these summaries would help them decide whether to join research studies and feel more informed about how their contributions matter. Why It Matters This tool could help patients and donors better understand research without needing a science degree. When people can see how studies work, they are more likely to participate in future research. While patient partners emphasized the need for summaries to be accurate and reliable, this approach shows promise as a unique strategy to better connect the public with research.
Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.
Comment cette classification a été obtenuedéplier
Prédiction machine sur la base complète
Imitation des enseignantsNi prévalence calibrée, ni vérité terrain. Validation humaine à venir. Le volet Gemma est une étiquette directe du modèle pour chaque travail de la base, lue sur la notice réduite au titre. Le volet Codex est un classifieur appris des 10 348 étiquettes directes de Codex et calibré sur les taux pondérés de l'échantillon; les champs sans appui suffisant ne portent aucun appel Codex. Le mode candidate est l'union des deux volets; le consensus est leur intersection. Ces sorties portent le statut machine_predicted_unvalidated et ne sont pas des étiquettes humaines.
Scores du classifieur distillé par catégorie (deux têtes)
| Catégorie | Codex | Gemma |
|---|---|---|
| Métarecherche | 0,014 | 0,106 |
| Méta-épidémiologie (sens strict) | 0,003 | 0,001 |
| Méta-épidémiologie (sens large) | 0,001 | 0,003 |
| Bibliométrie | 0,009 | 0,003 |
| Études des sciences et des technologies | 0,001 | 0,001 |
| Communication savante | 0,005 | 0,003 |
| Science ouverte | 0,001 | 0,004 |
| Intégrité de la recherche | 0,001 | 0,002 |
| Charge utile insuffisante (le modèle a refusé de juger) | 0,025 | 0,014 |
Scores machine (provisoires)
Les deux têtes enseignantes du modèle étudiant, lues sur ce travail. Un score ordonne la base pour la relecture; il n'affirme jamais une catégorie, et le statut de validation accompagne chaque rangée tel quel.
Scores de référence d'un modèle non mature (critères de maturité non atteints, 7 itérations). Un score ordonne; il n'affirme jamais une catégorie.
score_only:v0-immature-baseline · tel quel depuis la passe de notation : score_only signifie que le nombre peut ordonner les travaux, et qu'aucune étiquette de catégorie n'en découleClassification
machine, non validéePrédiction automatique; un appel candidat d’une seule source (Gemma direct ou Codex distillé), pas un consensus.
Le détail, modèle par modèle et score par score, se trouve en fin de page sous « Comment cette classification a été obtenue ».