Data extraction and comparison for complex systematic reviews: a step-by-step guideline and an implementation example using open-source software
Bibliographic record
Abstract
BACKGROUND: Data extraction (DE) is a challenging step in systematic reviews (SRs). Complex SRs can involve multiple interventions and/or outcomes and encompass multiple research questions. Attempts have been made to clarify DE aspects focusing on the subsequent meta-analysis; there are, however, no guidelines for DE in complex SRs. Comparing datasets extracted independently by pairs of reviewers to detect discrepancies is also cumbersome, especially when the number of extracted variables and/or studies is colossal. This work aims to provide a set of practical steps to help SR teams design and build DE tools and compare extracted data for complex SRs. METHODS: We provided a 10-step guideline, from determining data items and structure to data comparison, to help identify discrepancies and solve data disagreements between reviewers. The steps were organised into three phases: planning and building the database and data manipulation. Each step was described and illustrated with examples, and relevant references were provided for further guidance. A demonstration example was presented to illustrate the application of Epi Info and R in the database building and data manipulation phases. The proposed guideline was also summarised and compared with previous DE guidelines. RESULTS: The steps of this guideline are described generally without focusing on a particular software application or meta-analysis technique. We emphasised determining the organisational data structure and highlighted its role in the subsequent steps of database building. In addition to the minimal programming skills needed, creating relational databases and data validation features of Epi info can be utilised to build DE tools for complex SRs. However, two R libraries are needed to facilitate data comparison and solve discrepancies. CONCLUSIONS: We hope adopting this guideline can help review teams construct DE tools that suit their complex review projects. Although Epi Info depends on proprietary software for data storage, it can still be a potential alternative to other commercial DE software for completing complex reviews.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.432 | 0.106 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.065 | 0.003 |
| Bibliometrics | 0.001 | 0.003 |
| Science and technology studies | 0.001 | 0.000 |
| Scholarly communication | 0.007 | 0.002 |
| Open science | 0.009 | 0.002 |
| Research integrity | 0.001 | 0.000 |
| Insufficient payload (model declined to judge) | 0.001 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; both teacher heads agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".