315 - Reliability of AI Chatbots in Providing Urinary Tract Infection Health Information: A Comparative Study of ChatGPT, Google Gemini, and DeepSeek
Bibliographic record
Abstract
Hypothesis / aims of study This study aims to evaluate and compare the accuracy and completeness of responses generated by three AI models—ChatGPT, Gemini, and DeepSeek—when prompted with patient-oriented questions regarding female urinary tract infections. The findings will be measured against evidence-based clinical guidelines and publications. Study design, materials and methods A cross-sectional design was employed. Researchers developed five standardized, patient-focused questions on UTI management based on recent evidence and authoritative guidelines. Each question was individually submitted to ChatGPT, Gemini, and DeepSeek in a private browser session. Two medical professionals independently evaluated each AI-generated response for accuracy (1–3 scale: Correct, Partially Correct, or Incorrect) and completeness (1–2 scale: Incomplete or Complete). Both raters compared the AI response with the AUA guidelines. Inter-rater agreement was used to assess the consistency of ratings between evaluators. Results Regarding completeness, all three AI models received the highest score of 2 (Complete) from both raters in all five categorical questions. Rater 1 administered a score of 3 (Correct) for all three AI models as it pertains to accuracy in all five responses. Rater 2 provided DeepSeek with a score of 3 (Correct) in accuracy for all five responses. Rater 2 provided ChatGPT and Gemini with a score of 2 (Partially Correct) for their responses in the prevention category. Interpretation of results Inter-rater agreement was high across all models. Overall agreement for accuracy was 86.7%, while completeness ratings had 100% agreement. DeepSeek demonstrated the highest consistency, with 100% agreement between evaluators on both accuracy and completeness. ChatGPT and Gemini each showed 80% agreement for accuracy but maintained full agreement for completeness. Concluding message All three AI models produced generally accurate and complete responses to UTI-related patient questions. High inter-rater agreement, especially for completeness, suggests strong reliability of the content. However, small variations in accuracy ratings highlight the importance of consistent evaluation frameworks. DeepSeek demonstrated the highest overall consistency, indicating potential for reliable patient education support. Download: Download high-res image (105KB) Download: Download full-size image Figure 1 . Response Scores for ChatGPT, Gemini, and DeepSeek Funding NA Clinical Trial No Subjects None
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.075 | 0.262 |
| Meta-epidemiology (narrow) | 0.000 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.002 | 0.001 |
| Science and technology studies | 0.001 | 0.002 |
| Scholarly communication | 0.002 | 0.002 |
| Open science | 0.001 | 0.002 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.001 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".