ChatGPT compared with Google Search and healthcare institution as sources of postoperative patient instructions after gynecological surgery

The use of ChatGPT, an artificial intelligence language model, has the potential to improve healthcare delivery.1 The mode of delivering postoperative instructions has been shown to affect patients' actions and satisfaction.2, 3 As online search engines are common sources for patients' medical information,4 we aimed to study the value of ChatGPT-generated postoperative instructions for gynaecological procedures. We analysed postoperative patient instructions for five common gynaecological procedures—laparoscopic hysterectomy, laparoscopic myomectomy, hysteroscopy, colposcopy and dilatation and curettage. We obtained postoperative instructions from ChatGPT version 3.5, Google Search, and from our institution (Appendix S1). To generate instructions from ChatGPT and Google Search the following natural language sentence was entered: ‘Please provide postoperative instructions for a patient who just underwent a (procedure name)’. We used the first non-sponsored response result in Google Search for the analysis. Institutional resources used were standard, authentic hospital discharge instructions for the surgical procedures. We blinded the results by standardising fonts and removing any source identifiers. The primary outcome was the instructions’ Understandability and Actionability. We used the Patient Education Materials Assessment Tool–printable (PEMAT-P) to evaluate postoperative instructions.5 Two researchers, blinded to the source of the instructions, scored the instructions individually. The final scores are the average of both reviewers’ scores. Secondary outcomes included instruction word count, and whether detailed concerning signs and symptoms that should prompt urgent consultation were provided. We compared continuous parameters using Kruskal–Wallis one-way analysis of variance. Chi-square test was used to compare categorical parameters and a post-hoc test was performed. The Institutional Review Board waived the need for approval. Overall, understandability scores ranged from 70.9% to 91.7%, Actionability scores ranged from 60.0% to 80.0% and total PEMAT-P scores ranged from 67.7% to 88.3% (Table S1). ChatGPT scores ranged from 87.5% to 91.7% for Understandability, from 60% to 70% for Actionability, and from 79.4% to 82.4% for total PEMAT-P scores. In the comparison of ChatGPT with Google Search and Institution instructions, Understandability scores were similar across groups (Table 1). Actionability scores were significantly lower in the ChatGPT group compared with Institution instructions (median score 60.0 versus 80.0, respectively, p = 0.007). Total PEMAT-P scores were comparable among groups. Word count was significantly lower in ChatGPT instructions compared with Google Search (median 243.0 versus 658.0, respectively, p = 0.008). ChatGPT related less to concerning signs and symptoms when compared with Google search and Institution (20% versus 100% in both Google Search and Institution, p = 0.048). ChatGPT postoperative instructions scored lower in Actionability, underperformed in providing concerning signs and symptoms and generated approximately one-third of the text-length of the other sources studied. However, Understandability scores were similar across all postoperative instruction sources. ChatGPT was limited in providing patients with actionable instructions. These may include tangible tools such as checklists and specific medications instructions. Although short text length, as provided by ChatGPT, may provide succinct, targeted information, it is limited in providing specific actionable items. Moreover, important instructions regarding ‘red flags’ that should prompt an urgent consultation with a medical provider were lacking. Therefore, relying solely on ChatGPT's instructions may compromise patient safety. Despite these limitations, ChatGPT's strengths include its availability and ability to provide concise responses to specific medical questions that can be tailored for populations with different levels of cultural and social backgrounds. Limitations of the study include the number of procedures analysed, the use of only one set of instructions per procedure from the different sources, and the use of English language only. In addition, ChatGPT responses change with time and inputs, limiting the reproducibility of the study. ChatGPT may also provide incorrect information presented as genuine facts. Although we did not observe inaccuracies in our study, this should be acknowledged by users. In conclusion, a current ChatGPT version provided concise and clear postoperative instructions for specific gynecological procedures. RM, GL- conception, design, data acquisition, analysis and interpretation, manuscript drafting. MT, KW, MS, YB- data interpretation, critical revision of the manuscript draft. All authors approved the final version of the manuscript and are accountable for all aspects of the work. None. Cedars Sinai Medical Center, Los Angeles, CA, USA. MT is a consultant for Ethicon, Medtronic, Heracure Medical and Cooper Surgical; KW is a consultant for Aqua Therapeutics, Hologic, Ethicon and Karl Storz; MS is a consultant for Applied Medical and Intuitive Surgical; and RM is a consultant for Intuitive Surgical. All other authors—none declared. The institutional review board office determined that IRB review and approval is not required. The data that support the findings of this study are available from the corresponding author upon reasonable request. Appendix S1. Table S1. Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.