A115 THE PERFORMANCE OF NATURAL LANGUAGE PROCESSING IN INTERPRETING COLONOSCOPY REPORTS: A SYSTEMATIC REVIEW AND META-ANALYSIS
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
Abstract Background Screening colonoscopy is integral in the effort to identify and remove potentially cancerous lesions. Important quality indicators include the adenoma detection rate and more recently, the sessile/serrated adenoma detection rate. Natural language processing (NLP) is a computer-based linguistic technique that leverages artificial intelligence to abstract meaningful information from text. This tool carries the potential to automate the task of analyzing large volumes of colonoscopy and pathology reports to generate data on key performance metrics. Purpose The aim of this study is to systematically review the available literature on the performance of NLP in identifying the presence of an adenoma or a sessile/serrated adenoma in colonoscopy reports. Method We performed a systematic review and meta-analysis according to PRISMA recommendations. A comprehensive literature query was conducted on MEDLINE, EMBASE, CINAHL, and CDSR, through July 2022. Studies were included if they evaluated the performance of NLP in extracting data from colonoscopy reports. Our primary outcome was the performance of NLP models in correctly identifying an adenoma reported in a colonoscopy report. Two authors independently screened studies and abstracted data using an a priori designed data collection form. We pooled the sensitivity and specificity of our primary outcome using a univariate analysis first, followed by a bivariate analysis. Using the open-source package ‘mada’ which is written in R, we generated a summary estimate and a summary receiver operating characteristic curve. Result(s) From the 1030 unique studies obtained from our literature search, 13 studies met the inclusion criteria. Eligible studies were used for our meta-analysis. In the univariate analysis, the pooled sensitivity and specificity for detecting an adenoma by the NLP systems was 0.978 (95% CI 0.938-0.992) and 0.997 (95% CI 0.984-0.999), respectively. Similarly, in univariate analysis, the pooled sensitivity and specificity for detecting a sessile/serrated adenoma by the NLP systems was 0.984 (95% CI 0.929-0.996) and 1.0 (95% CI 0.998-1.000), respectively. In the bivariate analysis, the summary estimates for the sensitivity and specificity of the NLP system in detecting an adenoma were 0.973 (95% CI 0.929-0.990) and 0.992 (95%CI 0.978-0.997) respectively. For detecting a sessile/serrated adenoma, the summary estimates for sensitivity and specificity were 0.964 (95% CI 0.895-0.988) and 0.998 (95% CI 0.995-0.999) respectively. Conclusion(s) NLP models have excellent performance in extracting quality metric data from colonoscopy reports. Based on the available literature, we suggest integration of NLP in quality improvement efforts in colonoscopy. Please acknowledge all funding agencies by checking the applicable boxes below None Disclosure of Interest None Declared
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.011 | 0.003 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.003 | 0.001 |
| Bibliometrics | 0.001 | 0.001 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it