Use of Online Large Language Model Chatbots in Cornea Clinics
Bibliographic record
Abstract
PURPOSE: Online large language model (LLM) chatbots have garnered attention for their potential in enhancing efficiency, providing education, and advancing research. This study evaluated the performance of LLM chatbots-Chat Generative Pre-Trained Transformer (ChatGPT), Writesonic, Google Bard, and Bing Chat-in responding to cornea-related scenarios. METHODS: Prompts covering clinic administration, patient counselling, treatment algorithms, surgical management, and research were devised. Responses from LLMs were assessed by 3 fellowship-trained cornea specialists, blinded to the LLM used, using a standardized rubric evaluating accuracy, comprehension, compassion, professionalism, humanness, comprehensiveness, and overall quality. In addition, 12 readability metrics were used to further evaluate responses. Scores were averaged and ranked; subgroup analyses were performed to identify the best-performing LLM for each rubric criterion. RESULTS: Sixty-six responses were generated from 11 prompts. ChatGPT outperformed the other LLMs across all rubric criteria, scoring an overall response score of 3.35 ± 0.42 (83.8%). However, Google Bard excelled in readability, leading in 75% of the metrics assessed. Importantly, no responses were found to pose risks to patients, ensuring the safety and reliability of the information provided. CONCLUSIONS: ChatGPT demonstrated superior accuracy and comprehensiveness in responding to cornea-related prompts, whereas Google Bard stood out for its readability. The study highlights the potential of LLMs in streamlining various clinical, administrative, and research tasks in ophthalmology. Future research should incorporate patient feedback and ongoing data collection to monitor LLM performance over time. Despite their promise, LLMs should be used with caution, necessitating continuous oversight by medical professionals and standardized evaluations to ensure patient safety and maximize benefits.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.000 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".