Integration of Artificial Intelligence (AI) into the Data Extraction Phase of a Scoping Review
Bibliographic record
Abstract
This paper describes how artificial intelligence (AI) was used to assist with the data extraction phase of a scoping review, specifically comparing different AI models and the accuracy of AI-assisted data extraction compared to human extraction. Scoping reviews map existing literature on a topic and are useful for complex or under-reviewed subjects. Integrating AI, particularly large language models, can enhance processing speed and data analysis. Three models, ChatGPT 3.5 and -4 (both developed by OpenAI) and Copilot (by Microsoft), were compared to identify the best model for AI-assisted data extraction. Adobe Acrobat Pro’s Optical Character Recognition (OCR) feature and ‘ChatGPT Splitter’ were used to manage image-based content and large sections of data. A custom script was iteratively generated and implemented with the source material. AI-assisted extraction results were compared to text extracted by an independent reviewer. ChatGPT-4 was utilized to enhance efficiency and accuracy of data extraction from 234 sources. While human extraction was more specific with verbatim information, AI was faster and sometimes provided more nuanced understanding, averaging 20 minutes per source compared to one hour for human extraction. ChatGPT-4’s superior text processing capabilities made it the optimal choice. While AI advancements have streamlined data extraction, human oversight remains crucial to ensure accuracy and address biases. This methodology is especially beneficial for smaller research teams and emphasizes the importance of structured prompts and rigorous review. Careful planning and oversight can mitigate risks, ultimately improving the quality and efficiency of the review process.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.052 | 0.053 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.002 | 0.000 |
| Bibliometrics | 0.001 | 0.012 |
| Science and technology studies | 0.000 | 0.001 |
| Scholarly communication | 0.000 | 0.001 |
| Open science | 0.004 | 0.001 |
| Research integrity | 0.000 | 0.002 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; both teacher heads agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".