Record W2985923507 · doi:10.1038/s41586-020-2314-9

Variability in the analysis of a single neuroimaging dataset by many teams

2020· article· en· W2985923507 on OpenAlexafffund

Rotem Botvinik‐Nezer, Felix Holzmeister, Colin F. Camerer, Anna Dreber, Jürgen Huber, Magnus Johannesson, Michael Kirchler, Roni Iwanir, Jeanette A. Mumford, R. Alison Adcock, Paolo Avesani, Błażej M. Bączkowski, Aahana Bajracharya, Leah Bakst, Sheryl Ball, Marco Barilari, Nadège Bault, Derek Beaton, Julia Beitner, Roland G. Benoit, Ruud Berkers, Jamil P. Bhanji, Bharat B. Biswal, Sebastian Bobadilla-Suarez, Tiago Bortolini, Katherine L. Bottenhorn, Alexander Bowring, Senne Braem, Hayley R. Brooks, Emily G. Brudner, Cristian Buc Calderon, Julia A. Camilleri, Jaime J. Castrellon, Luca Cecchetti, Edna C. Cieslik, Zachary J. Cole, Olivier Collignon, Robert W. Cox, William A. Cunningham, Stefan Czoschke, Kamalaker Dadi, Charles P. Davis, Alberto De Luca, Mauricio R. Delgado, Lysia Demetriou, Jeffrey B. Dennison, Xin Di, Erin W. Dickie, Ekaterina Dobryakova, Claire Donnat, Juergen Dukart, Niall W. Duncan, Joke Durnez, Amr Eed, Simon B. Eickhoff, Andrew Erhart, Laura Fontanesi, G. Matthew Fricke, Shiguang Fu, Adriana Galván, Rémi Gau, Sarah Genon, Tristan Glatard, Enrico Glerean, Jelle J. Goeman, Sergej Golowin, Carlos González‐García, Krzysztof J. Gorgolewski, Cheryl L. Grady, Mikella A Green, João F. Guassi Moreira, Olivia Guest, Shabnam Hakimi, J. Paul Hamilton, Roeland Hancock, Giacomo Handjaras, Bronson Harry, Colin Hawco, Peer Herholz, Gabrielle Herman, Stephan Heunis, Felix Hoffstaedter, Jeremy Hogeveen, Susan Holmes, Hu Chuan-Peng, Scott A. Huettel, Matthew Hughes, Vittorio Iacovella, Alexandru D. Iordan, Peder Mortvedt Isager, Ayse Ilkay Isik, Andrew Jahn, Matthew R. Johnson, Tom Johnstone, Michael Joseph, Anthony Juliano, Joseph W. Kable, Michalis Kassinopoulos, Cemal Koba, Xiangzhen Kong, Timothy R. Koscik, Nuri Erkut Kucukboyaci, Brice A. Kuhl, Sebastian Kupek, Angela R. Laird, Claus Lamm, Robert Langner, Nina Lauharatanahirun, Hongmi Lee, Sangil Lee, Alexander Leemans, Andrea Leo, Elise Lesage, Flora Li, Monica Y. C. Li, Phui Cheng Lim, Evan Nathaniel Lintz, Schuyler Liphardt, Annabel B. Losecaat Vermeer, Bradley C. Love, Michael L. Mack, Norberto Malpica, Theo Marins, Camille Maumet, Kelsey McDonald, Joseph T. McGuire, Helena Melero, Adriana S. Méndez Leal, Benjamin Meyer, Kristin N. Meyer, Glad Mihai, Georgios D. Mitsis, Jorge Moll, Dylan M. Nielson, Gustav Nilsonne, Michael Notter, Emanuele Olivetti, Adrian Onicas, Paolo Papale, Kaustubh R. Patil, Jonathan E. Peelle, Alexandre Perez-Lebel, Doris Pischedda, Jean‐Baptiste Poline, Yanina Prystauka, Shruti Ray, Patricia A. Reuter‐Lorenz, Richard C. Reynolds, Emiliano Ricciardi, Jenny Rieck, Anais Rodriguez-Thompson, Anthony Romyn, Taylor Salo, Gregory R. Samanez‐Larkin, Emilio Sanz‐Morales, Margaret L. Schlichting, Douglas H. Schultz, Qiang Shen, Margaret A. Sheridan, Jennifer A. Silvers, Kenny Skagerlund, Alec Smith, David V. Smith, Peter Sokol‐Hessner, Simon R. Steinkamp, Sarah M. Tashjian, Bertrand Thirion, John Thorp, Gustav Tinghög, Loreen Tisdall, Steven Tompson, Claudio Toro‐Serey, Juan Jesús Torre, Leonardo Tozzi, Vuong Truong, Luca Turella, Anna van 't Veer, Tom Verguts, Jean M. Vettel, Sagana Vijayarajah, Khoi Vo, Matthew B. Wall, Wouter D. Weeda, Susanne Weis, David White, David Wisniewski, Alba Xifra‐Porxas, Emily A. Yearling, Sangsuk Yoon, Rui Yuan, Kenneth S.L. Yuen, Lei Zhang, Xu Zhang, Joshua Zosky, Thomas E. Nichols, Russell A. Poldrack, Tom Schönberg

Bibliographic record

VenueNature · 2020

Typearticle

Languageen

FieldNeuroscience

TopicFunctional Brain Connectivity Studies

Canadian institutionsMcGill UniversityCentre for Addiction and Mental HealthMontreal Neurological Institute and HospitalUniversity of TorontoConcordia UniversityBaycrest HospitalHealth Sciences Centre

FundersH2020 Marie Skłodowska-Curie ActionsNational Institutes of HealthMax-Planck-GesellschaftCanada First Research Excellence FundMinistry of Education, IndiaIsrael Science FoundationNational Institute of Biomedical Imaging and BioengineeringFonds Wetenschappelijk OnderzoekChina Postdoctoral Science FoundationMarcus och Amalia Wallenbergs minnesfondVlaamse regeringFonds De La Recherche Scientifique - FNRSAustrian Science FundNational Natural Science Foundation of ChinaVetenskapsrådetVienna Science and Technology FundHealth CanadaNederlandse Organisatie voor Wetenschappelijk OnderzoekWellcome TrustTel Aviv UniversityFondation Brain CanadaEuropean CommissionKnut och Alice Wallenbergs StiftelseNational Imaging FacilityNational Institute of Mental HealthUniversität BaselMcGill UniversityDeutsche ForschungsgemeinschaftNational Science Foundation

KeywordsWorkflowFlexibility (engineering)Computer scienceData scienceVariation (astronomy)Pipeline (software)Data miningStatistical powerStatisticsMathematics

DOI10.1038/s41586-020-2314-9

Abstract

fetched live from OpenAlex

No abstract in any covered source. Its absence is recorded, not treated as a negative.

No abstract. This is not a gap in this database; OpenAlex has none either. 23.3% of the frame is in this state, and the screen finds HALF as much metaresearch here, so the absence is a measured bias rather than a missing field.

How this classification was reachedexpand

Full frame machine prediction

Teacher imitation

Not calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.

metaresearch head score (Codex)0.065

metaresearch head score (Gemma)0.166

Version: metacan-v3-hybrid-931329e0061cValidation status: machine_predicted_unvalidated

Candidate categoriesMetaresearch

Consensus categoriesnone

DomainCandidate signal: Reproducibility · Consensus signal: none

Study designCandidate signal: Observational · Consensus signal: none

GenreCandidate signal: Empirical · Consensus signal: none

Teacher disagreement score0.935

Threshold uncertainty score0.343

Distilled classifier scores by category (both heads)

Category	Codex	Gemma
Metaresearch	0.065	0.166
Meta-epidemiology (narrow)	0.001	0.001
Meta-epidemiology (broad)	0.002	0.002
Bibliometrics	0.003	0.003
Science and technology studies	0.001	0.003
Scholarly communication	0.004	0.003
Open science	0.002	0.004
Research integrity	0.002	0.002
Insufficient payload (model declined to judge)	0.001	0.001

Machine scores (provisional)

The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.

Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.

Opus teacher head0.026

GPT teacher head0.279

Teacher spread0.253 · how far apart the two teachers sit on this one work

Validation statusscore_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it