Using GPT-4 to Automate the Generation of Lay Summaries for Cancer Publications
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
Abstract Background Cancer research literature is often riddled with technical jargon that is not digestible to the average person. Individuals interested in research studies may want to contribute through patient partner engagement or sample donation but find the relevant literature overwhelming. Through the generation of lay summaries, previously inaccessible research papers become easier to comprehend, especially for patient partners or data donors. With large language models (LLMs) continuing to advance, so does their capability to summarize large texts. Objectives In this study, we examined whether LLMs can produce lay summaries of scientific literature at-scale, while maintaining readability and accuracy to their source texts. Methods We developed a tool to generate lay summaries of open-access article abstracts and their full texts with GPT-4-Turbo. Prompt development aimed for a target 8th grade reading level assessed with Flesch-Kincaid Grade Level. Human-review metrics were used to evaluate readability and accuracy when generated using abstracts versus full text articles. Results The average Flesch-Kincaid Grade Level Score was 7.13 for abstract-based summaries and 7.39 for full text-based summaries, indicating summaries at around 7th grade reading level. Human-review metrics showed these summaries were of similar readability and accuracy when generated using abstracts versus full text articles, with mean accuracy scores from human review of 7.09 vs 7.42 out of 10 respectively. Additionally, qualitative patient-based assessment indicated these summaries would encourage participation in research studies. Conclusion By generating lay summaries for complex and lengthy research papers, their scientific information becomes accessible to a larger audience, including patient partners interested in contributing to cancer research. Summaries that are easy to understand will allow participants to make informed decisions about their involvement and appreciate the impact of their contributions if and when their results are published. Lay Summary This study explores if artificial intelligence (AI) can help make hard to read cancer research papers easier to understand for members of the public. Problem When people donate cancer tissue samples or participate in research studies, they often want to know how their contributions are being used. However, scientific papers are full of technical language that’s hard for most people to grasp. People in past studies have said this can make them less willing to take part in research. Methods The study created a computer program using AI (GPT-4-Turbo) to turn complex kidney cancer research papers into simple summaries. They tested whether the AI could summarize both short abstracts and full-length papers effectively. They aimed for summaries at a 6th to 8th-grade reading level. This was to follow Canadian and U.S. health communication guidelines. Results The AI created 106 summaries. Computer measures showed the summaries were close to a 7th-grade reading level. Though, researchers had to tell the AI to write for a 2nd-grade audience to achieve this. Of note, summaries from short abstracts were just as accurate and readable as those from full papers. Eighteen volunteers, including five patient partners, reviewed the summaries and rated them for clarity and accuracy. They were rated at around 7 out of 10 points. All patient partners said these summaries would help them decide whether to join research studies and feel more informed about how their contributions matter. Why It Matters This tool could help patients and donors better understand research without needing a science degree. When people can see how studies work, they are more likely to participate in future research. While patient partners emphasized the need for summaries to be accurate and reliable, this approach shows promise as a unique strategy to better connect the public with research.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.002 | 0.001 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.000 |
| Bibliometrics | 0.000 | 0.002 |
| Science and technology studies | 0.001 | 0.000 |
| Scholarly communication | 0.001 | 0.001 |
| Open science | 0.002 | 0.001 |
| Research integrity | 0.001 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it