Friend or Foe? The Role of Robots in Systematic Reviews
Notice bibliographique
Résumé
EditorialsJuly 2022Friend or Foe? The Role of Robots in Systematic ReviewsLisa Hartling, PhD and Allison Gates, PhDLisa Hartling, PhDAlberta Research Centre for Health Evidence, Department of Pediatrics, Faculty of Medicine & Dentistry, University of Alberta, Edmonton, Alberta, Canada and Allison Gates, PhDAlberta Research Centre for Health Evidence, Department of Pediatrics, Faculty of Medicine & Dentistry, University of Alberta, Edmonton, Alberta, CanadaAuthor, Article, and Disclosure Informationhttps://doi.org/10.7326/M22-1439 SectionsAboutFull TextPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissions ShareFacebookTwitterLinkedInRedditEmail The dramatic increase in publication of health literature has generated a growing need for evidence syntheses to support decision making. Most recently, the COVID-19 pandemic has created a remarkable and unprecedented demand for the rapid production of reliable evidence syntheses (1). Systematic reviewers recognize the need to create efficiencies in review production while maintaining methodological rigor to ensure valid conclusions. To this end, technologies (often supported by machine learning and artificial intelligence) are being developed and used to fully or partially automate various stages of the systematic review process (2).One such technology, RobotReviewer, was evaluated in a trial by ...References1. Global Commission on Evidence to Address Societal Challenges. The Evidence Commission report: a wake-up call and path forward for decision-makers, evidence intermediaries, and impact-oriented evidence producers. McMaster Health Forum; 2022. Google Scholar2. Khalil H, Ameen D, Zarnegar A. Tools to support the automation of systematic reviews: a scoping review. J Clin Epidemiol. 2022;144:22-42. [PMID: 34896236] doi:10.1016/j.jclinepi.2021.12.005 CrossrefMedlineGoogle Scholar3. Arno A, Thomas J, Wallace B, et al. Accuracy and efficiency of machine learning–assisted risk-of-bias assessments in “real-world” systematic reviews. A noninferiority randomized controlled trial. Ann Intern Med. 2022;175:1001-9. doi:10.7326/M22-0092 LinkGoogle Scholar4. Higgins JPT, Savović J, Page MJ, et al. Chapter 8: Assessing risk of bias in a randomized trial. In: Higgins JPT, Thomas J, Chandler J, et al, eds. Cochrane Handbook for Systematic Reviews of Interventions. Version 6.3 (updated February 2022). The Cochrane Collaboration; 2022. Google Scholar5. O’Connor AM, Tsafnat G, Thomas J, et al. A question of trust: can we build an evidence base to gain trust in systematic review automation technologies. Syst Rev. 2019;8:143. [PMID: 31215463] doi:10.1186/s13643-019-1062-0 CrossrefMedlineGoogle Scholar6. Schumi J, Wittes JT. Through the looking glass: understanding non-inferiority. Trials. 2011;12:106. [PMID: 21539749] doi:10.1186/1745-6215-12-106 CrossrefMedlineGoogle Scholar7. Soboczenski F, Trikalinos TA, Kuiper J, et al. Machine learning to help researchers evaluate biases in clinical trials: a prospective, randomized user study. BMC Med Inform Decis Mak. 2019;19:96. [PMID: 31068178] doi:10.1186/s12911-019-0814-z CrossrefMedlineGoogle Scholar8. Arno A, Elliott J, Wallace B, et al. The views of health guideline developers on the use of automation in health evidence synthesis. Syst Rev. 2021;10:16. [PMID: 33419479] doi:10.1186/s13643-020-01569-2 CrossrefMedlineGoogle Scholar9. Higgins JP, Altman DG, Gøtzsche PC, et al; Cochrane Bias Methods Group. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928. [PMID: 22008217] doi:10.1136/bmj.d5928 CrossrefMedlineGoogle Scholar10. Marshall IJ, Wallace BC. Toward systematic review automation: a practical guide to using machine learning tools in research synthesis [Editorial]. Syst Rev. 2019;8:163. [PMID: 31296265] doi:10.1186/s13643-019-1074-9 CrossrefMedlineGoogle Scholar Author, Article, and Disclosure InformationAffiliations: Alberta Research Centre for Health Evidence, Department of Pediatrics, Faculty of Medicine & Dentistry, University of Alberta, Edmonton, Alberta, CanadaNote: Dr. Gates is employed by the Canadian Agency for Drugs and Technologies in Health (CADTH). This work was unrelated to her employment, and CADTH had no role in the work reported. Drs. Hartling and Gates have collaborated on papers with Joanne McKenzie, an author of the RobotReviewer trial referenced in this editorial.Financial Support: Dr. Hartling is supported by a Canada Research Chair in Knowledge Synthesis and Translation.Disclosures: Disclosures can be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M22-1439.Corresponding Author: Lisa Hartling, PhD, Alberta Research Centre for Health Evidence, Department of Pediatrics, Faculty of Medicine & Dentistry, University of Alberta, 4-472 ECHA, 11405 87 Avenue, Edmonton, AB T6G 2J3, Canada; e-mail, lisa.[email protected]ca.This article was published at Annals.org on 31 May 2022. PreviousarticleNextarticle Advertisement FiguresReferencesRelatedDetailsSee AlsoAccuracy and Efficiency of Machine Learning–Assisted Risk-of-Bias Assessments in “Real-World” Systematic Reviews Anneliese Arno , James Thomas , Byron Wallace , Iain J. Marshall , Joanne E. McKenzie , and Julian H. Elliott Metrics July 2022Volume 175, Issue 7Page: 1045-1046KeywordsClinical epidemiologyMachine learningRandomized trialsResearch quality assessmentSystematic reviews ePublished: 31 May 2022 Issue Published: July 2022 Copyright & PermissionsCopyright © 2022 by American College of Physicians. All Rights Reserved.PDF downloadLoading ...
Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.
Comment cette classification a été obtenuedéplier
Prédiction machine sur la base complète
Imitation des enseignantsNi prévalence calibrée, ni vérité terrain. Validation humaine à venir. Le volet Gemma est une étiquette directe du modèle pour chaque travail de la base, lue sur la notice réduite au titre. Le volet Codex est un classifieur appris des 10 348 étiquettes directes de Codex et calibré sur les taux pondérés de l'échantillon; les champs sans appui suffisant ne portent aucun appel Codex. Le mode candidate est l'union des deux volets; le consensus est leur intersection. Ces sorties portent le statut machine_predicted_unvalidated et ne sont pas des étiquettes humaines.
Scores du classifieur distillé par catégorie (deux têtes)
| Catégorie | Codex | Gemma |
|---|---|---|
| Métarecherche | 0,556 | 0,884 |
| Méta-épidémiologie (sens strict) | 0,004 | 0,006 |
| Méta-épidémiologie (sens large) | 0,016 | 0,008 |
| Bibliométrie | 0,023 | 0,021 |
| Études des sciences et des technologies | 0,006 | 0,019 |
| Communication savante | 0,031 | 0,027 |
| Science ouverte | 0,014 | 0,011 |
| Intégrité de la recherche | 0,023 | 0,023 |
| Charge utile insuffisante (le modèle a refusé de juger) | 0,028 | 0,014 |
Scores machine (provisoires)
Les deux têtes enseignantes du modèle étudiant, lues sur ce travail. Un score ordonne la base pour la relecture; il n'affirme jamais une catégorie, et le statut de validation accompagne chaque rangée tel quel.
Scores de référence d'un modèle non mature (critères de maturité non atteints, 7 itérations). Un score ordonne; il n'affirme jamais une catégorie.
score_only:v0-immature-baseline · tel quel depuis la passe de notation : score_only signifie que le nombre peut ordonner les travaux, et qu'aucune étiquette de catégorie n'en découleClassification
machine, non validéePrédiction automatique; l’étiquette directe de Gemma et le classifieur distillé Codex s’accordent sur ce qui est montré ici.
Le détail, modèle par modèle et score par score, se trouve en fin de page sous « Comment cette classification a été obtenue ».