Calibration Feedback With the Practical Scoring Rule Does Not Improve Calibration of Confidence
Bibliographic record
Abstract
ABSTRACT People are often overconfident in their probabilistic judgments of future events or the state of their own knowledge. Some training methods have proven effective at reducing bias, but these usually involve intensive training sessions with experienced facilitators. This is not conducive to a scalable and domain‐general training program for improving calibration. In two experiments (N1 = 610, N2 = 871), we examined the effectiveness of a performance feedback calibration training paradigm based on the Practical scoring rule, a modification of the logarithmic scoring rule designed to be more intuitive to facilitate learning. We examined this training regime in comparison to a control group and an outcome feedback group. Participants were tasked with selecting which of two world urban agglomerations had a higher population and to provide their confidence level. The outcome feedback group received information about the correctness of their choice on a trial‐by‐trial basis as well as a summary of their percent correct after each experimental block. The performance feedback group received this information plus the Practical score on a trial‐by‐trial basis and information about their overall over‐ or underconfidence at the end of each block. We also examined whether Actively Open‐Minded Thinking (AOMT) was predictive of calibration and its change across blocks. We found no improvement in calibration due to either training regime. Good calibration overall was predicted by AOMT, but not its change across blocks. The results shed light on the generalizability of other findings showing positive effects of performance training using the Practical scoring rule.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.002 | 0.027 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.001 |
| Scholarly communication | 0.000 | 0.001 |
| Open science | 0.001 | 0.001 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.008 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".