Crowdsourcing the Citation Screening Process for Systematic Reviews: Validation Study
Why is this work in the frame?
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Full frame distilled prediction
Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
- Candidate categories
- Metaresearch, Meta-epidemiology (broad), Scholarly communication, Open science, Research integrity, Insufficient payload (model declined to judge)
- Consensus categories
- Metaresearch, Insufficient payload (model declined to judge)
- Domain
- Candidate signal: noneConsensus signal: none
- Study design
- Candidate signal: Not applicableConsensus signal: none
- Genre
- Candidate signal: ReviewConsensus signal: Review
- Teacher disagreement score
- 0.664
- Threshold uncertainty score
- 1.000
- Validation status
machine_predicted_unvalidated·codex-gemma-dda1882f352a
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.776 | 0.594 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.016 | 0.006 |
| Bibliometrics | 0.002 | 0.003 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.003 | 0.000 |
| Open science | 0.011 | 0.000 |
| Research integrity | 0.000 | 0.002 |
| Insufficient payload (model declined to judge) | 0.004 | 0.002 |
Machine scores (provisional)
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
- Teacher spread
- 0.227 · how far apart the two teachers sit on this one work
- Validation status
score_only:v0-immature-baseline· verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it
Abstract
BACKGROUND: Systematic reviews (SRs) are often cited as the highest level of evidence available as they involve the identification and synthesis of published studies on a topic. Unfortunately, it is increasingly challenging for small teams to complete SR procedures in a reasonable time period, given the exponential rise in the volume of primary literature. Crowdsourcing has been postulated as a potential solution. OBJECTIVE: The feasibility objective of this study was to determine whether a crowd would be willing to perform and complete abstract and full text screening. The validation objective was to assess the quality of the crowd's work, including retention of eligible citations (sensitivity) and work performed for the investigative team, defined as the percentage of citations excluded by the crowd. METHODS: We performed a prospective study evaluating crowdsourcing essential components of an SR, including abstract screening, document retrieval, and full text assessment. Using CrowdScreenSR citation screening software, 2323 articles from 6 SRs were available to an online crowd. Citations excluded by less than or equal to 75% of the crowd were moved forward for full text assessment. For the validation component, performance of the crowd was compared with citation review through the accepted, gold standard, trained expert approach. RESULTS: Of 312 potential crowd members, 117 (37.5%) commenced abstract screening and 71 (22.8%) completed the minimum requirement of 50 citation assessments. The majority of participants were undergraduate or medical students (192/312, 61.5%). The crowd screened 16,988 abstracts (median: 8 per citation; interquartile range [IQR] 7-8), and all citations achieved the minimum of 4 assessments after a median of 42 days (IQR 26-67). Crowd members retrieved 83.5% (774/927) of the articles that progressed to the full text phase. A total of 7604 full text assessments were completed (median: 7 per citation; IQR 3-11). Citations from all but 1 review achieved the minimum of 4 assessments after a median of 36 days (IQR 24-70), with 1 review remaining incomplete after 3 months. When complete crowd member agreement at both levels was required for exclusion, sensitivity was 100% (95% CI 97.9-100) and work performed was calculated at 68.3% (95% CI 66.4-70.1). Using the predefined alternative 75% exclusion threshold, sensitivity remained 100% and work performed increased to 72.9% (95% CI 71.0-74.6; P<.001). Finally, when a simple majority threshold was considered, sensitivity decreased marginally to 98.9% (95% CI 96.0-99.7; P=.25) and work performed increased substantially to 80.4% (95% CI 78.7-82.0; P<.001). CONCLUSIONS: Crowdsourcing of citation screening for SRs is feasible and has reasonable sensitivity and specificity. By expediting the screening process, crowdsourcing could permit the investigative team to focus on more complex SR tasks. Future directions should focus on developing a user-friendly online platform that allows research teams to crowdsource their reviews.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
The record
- Venue
- Journal of Medical Internet Research
- Topic
- Meta-analysis and systematic reviews
- Field
- Decision Sciences
- Canadian institutions
- BC Children's HospitalUniversity of OttawaChildren's Hospital of Eastern OntarioUniversity of British Columbia
- Funders
- CHEO Research Institute
- Keywords
- CrowdsourcingCitationComputer scienceData scienceProcess (computing)Systematic reviewMEDLINEWorld Wide WebPolitical science
- Has abstract in OpenAlex
- yes