MétaCan
Menu
Back to cohort
Record W2733427969 · doi:10.1097/ede.0000000000000706

Re

2017· letter· war· W2733427969 on OpenAlex
Michael A. McIsaac

Why this work is in the frame

A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.

affAt least one author lists a Canadian institution in the pinned OpenAlex snapshot.
aboutThe title or abstract carries a Canadian signal from the geographic lexicon.

Bibliographic record

VenueEpidemiology · 2017
Typeletter
Languagewar
FieldComputer Science
TopicBayesian Modeling and Causal Inference
Canadian institutionsQueen's University
Fundersnot available
KeywordsPropositionBayes' theoremBelief revisionMistakeEpistemologySet (abstract data type)Point (geometry)Mathematical economicsComputer sciencePsychologyMathematicsPhilosophyArtificial intelligenceBayesian probabilityLawPolitical science

Abstract

fetched live from OpenAlex

To the Editor: In their letter, Cole et al1 set out to prove that a dogmatist (one whose beliefs cannot be influenced by observations) cannot learn (which they define as “having your beliefs influenced by observations”). Disregarding that this proposition is true by definition, the authors provided an amusing exploration into this idea using Bayes’ theorem. However, the authors made an important error in their calculations. This error does not change the authors’ point about dogmatists being unable to learn, but it does lead to the incorrect conclusion that it is good practice to allow any piece of information to sway your opinion, regardless of the quality of that information; this seems like a major oversight in this era of “fake news”2 and “alternative facts!”3,4 Cole et al1 consider an encrypted question that has two possible answers; we will denote these as or . The authors suppose that 12 people answer the question of interest and examine how prior belief in the correct answer is updated by these observations. The authors remind us of Bayes’ theorem, which says that belief in an answer after observing data [the posterior probability ] is a function of both prior belief and the likelihood of the observed data if the answer is . They define a dogmatic belief as one that is certain a priori [e.g., ] and observe that data do not influence this dogmatic belief . However, in their calculations, Cole et al1 conflated their answer to the question of interest with the probability that each person actually gave the right answer. In the remainder of this letter, I will correct the calculations and demonstrate the important finding that learning from bad information can be worse than not learning at all. As in reference (1), suppose that in absence of evidence, a nondogmatist believes the two possible answers to be equally likely . Cole et al1 wish to calculate the nondogmatist’s belief that after learning that five people think and seven think . That is, the goal is to find the posterior probability. The key point missed by Cole et al1 is that is a function of the probability (call it ) that each of these independent and exchangeable people gave the right answer. In their example with deterministic Q, If the observations came from 12 individuals who were each fairly likely to get the right answer, say = 2/3, then the posterior probabilities would be and , and as reported by Cole et al,1 a nondogmatist should be swayed to believe that the correct answer is . However, if the 12 “experts” were answering completely randomly (so = 1/2), then we can see that their information truly should not influence beliefs: and . Most importantly, if the information came from 12 people who were unlikely to give the right answer, say = 1/3, then beliefs should actually be influenced in the opposite direction: and . Without knowing the quality of the data, it is impossible to properly learn from it. If presented with bad data, the nondogmatist could easily be swayed to posterior beliefs that are further from the truth than the ones established a priori! In this light, I propose an addendum to the proposition by Cole et al1: it is true that dogmatists cannot learn, but learning may actually be detrimental to those who cannot separate truth from “alternative facts.” Michael A. McIsaac Department of Public Health Sciences Queen’s University Kingston, ON, Canada [email protected]

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.

Full frame distilled prediction

Teacher imitation

Not calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.

metaresearch head score (Codex)0.005
metaresearch head score (Gemma)0.006
Version: codex-gemma-dda1882f352aValidation status: machine_predicted_unvalidated
Candidate categoriesMeta-epidemiology (narrow), Open science, Research integrity, Insufficient payload (model declined to judge)
Consensus categoriesResearch integrity
DomainCandidate signal: none · Consensus signal: none
Study designCandidate signal: Not applicable · Consensus signal: Not applicable
GenreCandidate signal: Commentary · Consensus signal: Commentary
Teacher disagreement score0.375
Threshold uncertainty score0.999

Codex and Gemma teacher scores by category

CategoryCodexGemma
Metaresearch0.0050.006
Meta-epidemiology (narrow)0.0010.001
Meta-epidemiology (broad)0.0020.001
Bibliometrics0.0000.000
Science and technology studies0.0010.001
Scholarly communication0.0000.000
Open science0.0060.001
Research integrity0.0030.004
Insufficient payload (model declined to judge)0.0000.005

Machine scores (provisional)

The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.

Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.

Opus teacher head0.197
GPT teacher head0.379
Teacher spread0.182 · how far apart the two teachers sit on this one work
Validation statusscore_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it