Record W2594171765 · doi:10.1016/j.annemergmed.2016.12.025

Individual Gestalt Is Unreliable for the Evaluation of Quality in Medical Education Blogs: A METRIQ Study

2017· article· en· W2594171765 on OpenAlexaff

Brent Thoma, Stefanie S. Sebok‐Syer, Keeth Krishnan, Marshall Siemens, N. Seth Trueger, Isabelle N Colmers-Gray, Rob Woods, Emil Petrusa, Teresa M. Chan, Charlotte Alexander, Mohammed Alkhalifah, Saeed Alqahtani, Scott Anderson, Shelaina Anderson, Colin Andrews, Jocelyn Andruko, Felix Ankel, Nikytha Antony, Diptesh Aryal, Barbra E. Backus, Jennifer Baird, Andrew Baker, Sarah Batty, Jared Baylis, Braeden Beaumont, Chris Belcher, Brent Benavides, Michael D. Benham, Elyse Berger Pelletier, Julian Botta, Nicholas Bouchard, Victoria Brazil, Emily Brumfield, Anthony Bryson, Wisarut Bunchit, Kat Butler, Lindy Buzikievich, David Calcara, Rob Carey, Stephen Carroll, Casey Lyons, Louise Cassidy, Kirsty Challen, Timothy Chaplin, Natasha Chatham-Zvelebil, Eric Chen, Lucy Chen, Sushant Chhabra, Alvin Chin, Eric Chochi, Tina Choudhri, Jeremy Christensen, Kimberly Connors, Veronica Coppersmith, Abby Cosgrove, Gregory Costello, Kevin Cullison, Andrew D. D’Alessandro, Kerstin de Wit, Marie Decock, Rayan Delbani, William Denq, Julianna Deutscher, Brendan Devine, Maia Dorsett, Taylor Duda, Justin Dueweke, Teresa Dunphy, Sean Dyer, Kathryn Eastley, Marcia L. Edmonds, Ken Edwards, Robert R. Ehrman, Youness Elkhalidy, Preston J. Fedor, Brian Ficiur, Caley Flynn, Bill Fraser, Meagan Fu, James Fukakusa, Eric Funk, Damjan Gaco, Viktor Gawlik, Kenn Ghaffarian, Laleh Gharahbaghian, Phil Griffith, Andrew Griffith, Andrew Grock, Tanner Gronowski, Cathy Grossman, Jaroslaw Gucwa, Pawan Kumar Gupta, Alexandra Gustafson, Andrew Guy, Mary R. Haas, Stanislaw Haciski, Emina Hajdinjak, Andrew K. Hall, Regina Hammock, Jan Hansel, Alexander Hart, Larissa Hattin, Brandon Herb, SueLin Hilbert, Jesse Hill, Jeff Hill, Emily House, Nina House, James L. Huffman, Alex Ireland, Mohammed Ali Jamal, Victor Jansen, Zach Jarou, Vivian Jia, Levi Johnston, Drew Kalnow, Puneet Kapur, Seth Kelly, Kyle Kelson, William Kent, Rishi Khakhkhar, Jaasmit Khurana, Ashley Kilp, Scott Knapp, Sebastian Köhler, Ivanna Kruhlak, Nadim Lalani, Samantha Lam, Patrick M. Lank, Zander Laurie, Kristina Lea, Ernest Leber, Ching-Hsing Lee, Haakon Lenes, Nilantha Lenora, Jesse Leontowicz, Kelly Lien, Ying‐Chun Lin, Michelle Lin, Andrew Little, Ivy Liu, Harry Liu, Steve Liu, Stephanie Louka, Elise Lovell, David Lowe, Ashley Lubberdink, Jessica G.Y. Luc, Sheng‐Hsiang Ma, Hugh MacLeod, Nick Mancuso, Anali Maneshi, Dra. Maria Rosa Carrillo, Jesse May, John Mayo, M. M. McDonnell, Susan McLellan, Carolyn McQuarrie, Julia Nood, Therese Mead, Cory Meeuwisse, Patrick Meloy, Perry J. C. Menzies, Anne Messman, Stephen Miazga, Logan Mills, Ken Milne, Allan Mix, Steve Montag, Brendon Moore, Justin Morgenstern, Sarah Mott, P. Mukherj, Ali Mulla, Sheena Nandalal, Taylor Nikel, Sean Nugent, Morgan Oakland, Werner Oberholzer, Onyeka Otugo, Taofiq Segun Oyedokun, Mike Paddock, Alim Pardhan, Kinjal Patel, Quinten S. Paterson, Catherine Patocka, Christine Patterson, James Pearlman, Alexis Pelletier‐Bui, Marc Phan, Zafrina Poonja, Aubrey Powell, Kamini Premkumar, Gregor Prosen, Vishal Puri, Tanis Quaife, Ryan Raffel, Ali S. Raja, Randi Ramunno, Louise Rang, Suzanne Rannazzisi, Shauna Regan, Milan L. Ridderikhof, Vanessa L. Rogers, Christine Roh, Keith Rosenberg, Marina Roure, Sherri L Rudinsky, Joshua Rudner, Adeeb Saleh, Will Sanderson, Owen Scheirer, Paul N. Schofield, Paul Schunk, Evan S. Schwarz, Parisa Shahrabadi, Eric Shappell, Julia M. Sheffield, Jonathan Sherbino, Manpreet Singh, Hector C. Singson, Dave Slessor, Sam Smith, Paula Sneath, Robert Sobehart, Kerry Spearing, James Stempien, Britni Sternard, Tara Stratton, Katherine Stuart, Bob Stuntz, Michael Susalla, Colleen Sweeney, Loice Swisher, Henry Swoboda, Shahbaz Syed, Taku Taira, Nikhil Tambe, Richard Tang, Elisha Targonsky, Rachel Taylor, Alan G. Taylor, Todd B. Taylor, Paxton Ting, Gerhard Tiwald, Kelvin Tran, Evelyn Tran, Jason Trickovic, Paul Trinquero, Seth Trueger, Aaron Tyagi, Manrique Umana, Patrick Vallance, Patricia van den Berg, Luis Vargas, René Verbeek, Sandra Viggers, Zlata Vlodaver, M Wagner, Noorin Walji, Joe Walter, Miranda Wan, Rachel Wang, Gregory K. Wanner, Wyatt Warawa, Mike Ward, Jennifer Weekes, Kristen Weersink, Cara Weessies, Anna Whalen-Browne, Brian Whiteside, Matthew Willis, Jonas Wilmer, Nelson Wong, Mark W. Woodcroft, Lawrence Yau, Jessica Yee, Calvin H. Yeh, Simon Huang, Katherine Yurkiw, Fareen Zaver, Alexander Zozula

Bibliographic record

VenueAnnals of Emergency Medicine · 2017

Typearticle

Languageen

FieldPsychology

TopicChild Therapy and Development

Canadian institutionsMcMaster UniversityUniversity of TorontoWestern UniversityUniversity of AlbertaUniversity of Saskatchewan

Fundersnot available

KeywordsGeneralizability theoryIntraclass correlationGestalt psychologyMedicineLikert scalePearson product-moment correlation coefficientReliability (semiconductor)CorrelationFamily medicineClinical psychologyStatisticsPsychometricsPsychology

DOI10.1016/j.annemergmed.2016.12.025

Abstract

fetched live from OpenAlex

No abstract in any covered source. Its absence is recorded, not treated as a negative.

No abstract. This is not a gap in this database; OpenAlex has none either. 23.3% of the frame is in this state, and the screen finds HALF as much metaresearch here, so the absence is a measured bias rather than a missing field.

How this classification was reachedexpand

Full frame machine prediction

Teacher imitation

Not calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.

metaresearch head score (Codex)0.032

metaresearch head score (Gemma)0.219

Version: metacan-v3-hybrid-931329e0061cValidation status: machine_predicted_unvalidated

Candidate categoriesMetaresearch

Consensus categoriesnone

DomainCandidate signal: Evaluation · Consensus signal: none

Study designCandidate signal: Observational · Consensus signal: Observational

GenreCandidate signal: Empirical · Consensus signal: Empirical

Teacher disagreement score0.968

Threshold uncertainty score0.167

Distilled classifier scores by category (both heads)

Category	Codex	Gemma
Metaresearch	0.032	0.219
Meta-epidemiology (narrow)	0.000	0.000
Meta-epidemiology (broad)	0.001	0.001
Bibliometrics	0.003	0.002
Science and technology studies	0.002	0.003
Scholarly communication	0.004	0.005
Open science	0.001	0.003
Research integrity	0.002	0.002
Insufficient payload (model declined to judge)	0.005	0.001

Machine scores (provisional)

The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.

Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.

Opus teacher head0.476

GPT teacher head0.596

Teacher spread0.120 · how far apart the two teachers sit on this one work

Validation statusscore_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it