Validation of a generative artificial intelligence tool for the critical appraisal of articles on the epidemiology of mental health: Its application in the Middle East and North Africa
Bibliographic record
Abstract
Mental health disorders have a high disability-adjusted life years in the Middle East and North Africa. This rise has led to a surge in related publications, prompting researchers to use AI tools like ChatGPT to reduce time spent on routine tasks. Our study aimed to validate an AI-assisted critical appraisal (CA) tool by comparing it with human raters. We developed customized GPT models using ChatGPT-4. These models were tailored to evaluate studies using the Newcastle-Ottawa Scale (NOS) or the Jadad Scale in one model, while another model evaluated STROBE or CONSORT guidelines. Our results showed a moderate to good agreement between human CA and our GPTs for the NOS for cohort, case control and cross-sectional studies and for the Jadad scale, with an ICC of 0.68 [95 %CI: 0.24-0.82], 0.69 [95 %CI: 0.31-0.88], 0.76 [95 %CI: 0.47-0.90] and 0.84 [95 %CI: 0.57-0.94] respectively. There was also a moderate to substantial agreement between the two methods for STROBE in cross sectional, cohort, case control studies, and for CONSORT in trial design, with a K of 0.63 [95 %CI: 0.56-0.70], 0.57 [95 %CI: 0.47-0.66], 0.48 [95 %CI: 0.38-0.50] and 0.70 [95 %CI: 0.63-0.77] respectively. Our custom GPT models produced hallucinations in 6.5 % and 4.9 % of cases, respectively. Human raters took an average of 19.6 ± 4.3 min per article, whereas our customized GPTs took only 1.4. ChatGPT could be a useful tool for handling repetitive tasks yet its effective application relies on the critical expertise of researchers.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.010 | 0.016 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".