← Back to cohort
Nonstandard Errors
Albert J. Menkveld, Anna Dreber, Felix Holzmeister, Jürgen Huber, Magnus Johannesson, Michael Kirchler, Sebastian Neusüss, Michael Razen, Utz Weitzel, DAVID ABAD‐DÍAZ, Menachem Abudy, Tobias Adrian, Yacine Aı̈t-Sahalia, Olivier Akmansoy, Jamie Alcock, Vitali Alexeev, Arash Aloosh, Livia Amato, Diego Amaya, James J. Angel, ALEJANDRO T. AVETIKIAN, Amadeus Bach, Edwin Baidoo, Gaetan Bakalli, Bao Li, Andrea Barbon, Oksana Bashchenko, Parampreet Christopher Bindra, Geir Høidal Bjønnes, Jeffrey R. Black, Bernard S. Black, Dimitar Bogoev, SANTIAGO BOHORQUEZ CORREA, Oleg Bondarenko, Charles S. Bos, Ciril Bosch-Rosa, Elie Bouri, Christian T. Brownlees, Anna Calamia, Viet Nga Cao, Gunther Capelle‐Blancard, LAURA M. CAPERA ROMERO, Massimiliano Caporin, Allen Carrion, Tolga Caskurlu, Bidisha Chakrabarty, Jian Chen, Mikhail Chernov, William Cheung, Ludwig B. Chincarini, Tarun Chordia, SHEUNG‐CHI CHOW, Benjamin Clapham, Jean-Édouard Colliard, Carole Comerton‐Forde, Edward T. Curran, Thông Dao, Wale Dare, Ryan J. Davies, Riccardo De Blasis, GIANLUCA F. DE NARD, Fany Declerck, Oleg Deev, Hans Degryse, Solomon Y. Deku, Christophe Desagre, Mathijs A. van Dijk, Chukwuma Dim, Thomas Dimpfl, Yun Jiang Dong, P. Drummond, Tom L. Dudda, Teodor Duevski, Ariadna Dumitrescu, Teodor Dyakov, Anne Haubo Dyhrberg, Michał Dzieliński, Asli Eksi, Izidin El Kalak, Saskia ter Ellen, Nicolas Eugster, Martin D.D. Evans, Michael Farrell, ESTER FELEZ‐VINAS, Gerardo Ferrara, El Mehdi Ferrouhi, Andrea Flori, Jonathan Fluharty-Jaidee, Sean Foley, Kingsley Y. L. Fong, Thierry Foucault, Tatiana Franus, Francesco A. Franzoni, Bart Frijns, Michael Frömmel, SERVANNA M. FU, Sascha Füllbrunn, Baoqing Gan, Ge Gao, Thomas Gehrig, Roland Gemayel, Dirk Gerritsen, Javier Gil‐Bazo, Dudley Gilder, Lawrence R. Glosten, Thomas M. Gomez, Arseny Gorbenko, Joachim Grammig, Vincent Grégoire, Ufuk Güçbilmez, Björn Hagströmer, Julien Hambuckers, Erik Hapnes, Jeffrey H. Harris, Lawrence Harris, Simon Hartmann, Jean‐Baptiste Hasse, Nikolaus Hautsch, Xue‐Zhong He, Davidson Heath, Simon Hediger, Terrence Hendershott, Ann Marie Hibbert, Erik Hjalmarsson, SETH A. HOELSCHER, Peter Hoffmann, Craig W. Holden, Alex R. Horenstein, Wenqian Huang, Da Huang, Christophe Hurlin, Konrad Ilczuk, Alexey Ivashchenko, Subramanian R. Iyer, Hossein Jahanshahloo, Naji Jalkh, Charles M. Jones, Simon Jurkatis, Petri Jylhä, Andreas Kaeck, G. Kaiser, Arzé Karam, Egle Karmaziene, Bernhard Kassner, Markku Kaustia, E. S. Kazak, Fearghal Kearney, Vincent van Kervel, Saad Ahmed Khan, MARTA K. KHOMYN, Tony Klein, Olga A. Klein, Alexander Klos, Michael Koetter, Aleksey Kolokolov, Robert A. Korajczyk, Roman Kozhan, Jan Pieter Krahnen, Paul Kuhle, Amy Dict-Weng Kwan, Quentin Lajaunie, F.Y. Eric C. Lam, Marie Lambert, Hugues Langlois, Jens Lausen, Tobias Lauter, Markus Leippold, Vladimir Levin, Yijie Li, Hui Li, Chee Yoong Liew, Thomas Lindner, Oliver Linton, Jiacheng Liu, Anqi Liu, Guillermo Llorente, Matthijs Lof, Ariel Lohr, Francis A. Longstaff, Alejandro Lopez-Lira, Shawn Mankad, Nicola Mano, Alexis Marchal, Charles Martineau, Francesco Mazzola, Debrah Meloso, MICHAEL G. MI, Roxana Mihet, V. Mohan, Sophie Moinas, David Moore, Liangyi Mu, Dmitriy Muravyev, DERMOT MURPHY, Gábor Neszveda, Christian Neumeier, Ulf Nielsson, Mahendrarajah Nimalendran, Sven Nolte, Lars L. Nordén, Peter O’Neill, Khaled Obaid, Bernt Arne Ødegaard, Per Östberg, Emiliano Pagnotta, Marcus Painter, Stefan Palan, IMON J. PALIT, Andreas Park, Roberto Pascual, Paolo Pasquariello, Ľuboš Pástor, Vinay Patel, Andrew J. Patton, Neil D. Pearson, Loriana Pelizzon, Michele Pelli, Matthias Pelster, Christophe Pérignon, Cameron Pfiffer, Richard Philip, Tomáš Plíhal, Puneet Prakash, Oliver-Alexander Press, Tina Prodromou, Marcel Prokopczuk, Tālis J. Putniņš, Ya Qian, Gaurav Raizada, David A. Rakowski, Angelo Ranaldo, Luca Regis, Stefan Reitz, Thomas Renault, REX W. RENJIE, Roberto Renò, Steven Riddiough, Kalle Rinne, PAUL RINTAMÄKI, Ryan Riordan, Thomas Rittmannsberger, Inaki Rodriguez Longarela, DOMINIK ROESCH, Lavinia Rognone, Brian Roseman, Ioanid Roşu, Saurabh Roy, Nicolas Rudolf, Stephen Rush, Khaladdin Rzayev, Aleksandra Rzeźnik, Anthony J. Sanford, Harikumar Sankaran, Asani Sarkar, Lucio Sarno, Olivier Scaillet, Stefan Scharnowski, Klaus Reiner Schenk–Hoppé, Andrea Schertler, Michael Schneider, Florian Schroeder, Norman Schürhoff, Philipp Schuster, Marco A. Schwarz, Mark S. Seasholes, Norman Seeger, Or Shachar, Andriy Shkilko, Jessica Shui, Mario Šikić, Giorgia Simion, Lee A. Smales, Paul Söderlind, Elvira Sojli, Konstantin Sokolov, Jantje Sönksen, Laima Spokeviciute, Denitsa Stefanova, Marti G. Subrahmanyam, Barnabás Szászi, Oleksandr Talavera, Yuehua Tang, Nick Taylor, Wing Wah Tham, Erik Theissen, Julian Thimme, Ian Tonks, Hai Tran, Luca Trapin, Anders B. Trolle, Maria Văduva, Giorgio Valente, Robert A. Van Ness, Aurelio Vasquez, Thanos Verousis, Patrick Verwijmeren, Anders Vilhelmsson, Grigory Vilkov, VLADIMIR VLADIMIROV, Stefan Voigt, Wolf Wagner, Thomas Walther, Patrick Weiß, Michel van der Wel, Ingrid M. Werner, P. Joakim Westerholm, Christian Westheide, Hans C. Wika, Evert Wipplinger, Michael Wolf, Christian C. P. Wolff, Leonard Wolk, Wing‐Keung Wong, Jan Wrampelmeyer, Zhenhua Wu, Shuo Xia, Dacheng Xiu, KE XU, Caihong Xu, Pradeep Yadav, José Yagüe, Cheng Yan, Antti Yang, Woongsun Yoo, Wenjia Yu, Yihe Yu, Shihao Yu, Bart Zhou Yueshen, Darya Yuferova, Marcin Zamojski, Abalfazl Zareei, Stefan Zeisberger, Lu Zhang, S. Sarah Zhang, Xiaoyu Zhang, Lu Zhao, Zhuo Zhong, Z. Ivy Zhou, Chen Zhou, XINGYU S. ZHU, Marius Zoican, Remco C. J. Zwinkels
Bibliographic record
VenueThe Journal of Finance · 2024
Typearticle
Languageen
FieldMathematics
TopicMathematical and Theoretical Analysis
Canadian institutionsWilfrid Laurier UniversityHEC MontréalCanadian Nautical Research Society
FundersBooth School of Business, University of ChicagoLeonard N. Stern School of Business, New York UniversityUniversität LeipzigChina Medical UniversityUniversität MannheimUniversität ZürichLeibniz-GemeinschaftUniversidad Carlos III de MadridAsia UniversityStockholms UniversitetRiksbankens JubileumsfondNederlandse Organisatie voor Wetenschappelijk OnderzoekUniversität WienEberhard Karls Universität TübingenErasmus Universiteit RotterdamEötvös Loránd TudományegyetemUniversité du LuxembourgUniversiteit van AmsterdamUniversidad de MurciaAgence Nationale de la RechercheKnut och Alice Wallenbergs StiftelseNew York University ShanghaiUniversity of BristolCardiff UniversityHáskólinn í ReykjavíkLunds UniversitetHang Seng University of Hong KongUniversità di BolognaCopenhagen Business SchoolUniversiteit UtrechtLoyola Marymount UniversityUniversity of MinnesotaWilfrid Laurier UniversityUniversity of EssexZhongnan University of Economics and LawTechnische Universität DresdenUniversity of MemphisVrije Universiteit AmsterdamTrường Đại học Kinh tế - Luật, Đại học Quốc gia Thành phố Hồ Chí MinhChina Medical University HospitalUniversity of OklahomaUniversity of New South WalesAustrian Science FundOhio State UniversityUniversität St. GallenArizona State University
KeywordsComputer science
Abstract
ABSTRACT In statistics, samples are drawn from a population in a data‐generating process (DGP). Standard errors measure the uncertainty in estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence‐generating process (EGP). We claim that EGP variation across researchers adds uncertainty—nonstandard errors (NSEs). We study NSEs by letting 164 teams test the same hypotheses on the same data. NSEs turn out to be sizable, but smaller for more reproducible or higher rated research. Adding peer‐review stages reduces NSEs. We further find that this type of uncertainty is underestimated by participants.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
metaresearch head score (Codex)0.126
metaresearch head score (Gemma)0.463
Version: metacan-v3-hybrid-931329e0061cValidation status: machine_predicted_unvalidated
Candidate categoriesnone
Consensus categoriesnone
DomainCandidate signal: none · Consensus signal: none
Study designCandidate signal: Theoretical or conceptual · Consensus signal: Theoretical or conceptual
GenreCandidate signal: Methods · Consensus signal: Methods
Teacher disagreement score0.126
Threshold uncertainty score0.669
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.126 | 0.463 |
| Meta-epidemiology (narrow) | 0.003 | 0.001 |
| Meta-epidemiology (broad) | 0.004 | 0.004 |
| Bibliometrics | 0.005 | 0.007 |
| Science and technology studies | 0.002 | 0.009 |
| Scholarly communication | 0.006 | 0.005 |
| Open science | 0.007 | 0.006 |
| Research integrity | 0.005 | 0.008 |
| Insufficient payload (model declined to judge) | 0.021 | 0.010 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
Opus teacher head0.025
GPT teacher head0.314
Teacher spread0.289 · how far apart the two teachers sit on this one work
Validation status
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
The models applied no category: nothing in the taxonomy fit this work.
Study designTheoretical or conceptual
Domainnot available
GenreMethods
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".
Quick stats
Citations92
Published2024
Admission routes2
Has abstractyes