Utility of ChatGPT for Automated Creation of Patient Education Handouts: An Application in Neuro-Ophthalmology
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
BACKGROUND: Patient education in ophthalmology poses a challenge for physicians because of time and resource limitations. ChatGPT (OpenAI, San Francisco) may assist with automating production of patient handouts on common neuro-ophthalmic diseases. METHODS: We queried ChatGPT-3.5 to generate 51 patient education handouts across 17 conditions. We devised the "Quality of Generated Language Outputs for Patients" (QGLOP) tool to assess handouts on the domains of accuracy/comprehensiveness, bias, currency, and tone, each scored out of 4 for a total of 16. A fellowship-trained neuro-ophthalmologist scored each passage. Handout readability was assessed using the Simple Measure of Gobbledygook (SMOG), which estimates years of education required to understand a text. RESULTS: The QGLOP scores for accuracy, bias, currency, and tone were found to be 2.43, 3, 3.43, and 3.02 respectively. The mean QGLOP score was 11.9 [95% CI 8.98, 14.8] out of 16 points, indicating a performance of 74.4% [95% CI 56.1%, 92.5%]. The mean SMOG across responses as 10.9 [95% CI 9.36, 12.4] years of education. CONCLUSIONS: The mean QGLOP score suggests that a fellowship-trained ophthalmologist may have at-least a moderate level of satisfaction with the write-up quality conferred by ChatGPT. This still requires a final review and editing before dissemination. Comparatively, the rarer 5% of responses collectively on either extreme would require very mild or extensive revision. Also, the mean SMOG score exceeded the accepted upper limits of grade 8 reading level for health-related patient handouts. In its current iteration, ChatGPT should be used as an efficiency tool to generate an initial draft for the neuro-ophthalmologist, who may then refine the accuracy and readability for a lay readership.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.001 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.000 |
| Bibliometrics | 0.001 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it