Effect of Blinded Peer Review on Abstract Acceptance
Bibliographic record
Abstract
CONTEXT: Peer review should evaluate the merit and quality of abstracts but may be biased by geographic location or institutional prestige. The effectiveness of blinded peer review at reducing bias is unknown. OBJECTIVE: To evaluate the effect of blinded review on the association between abstract characteristics and likelihood of abstract acceptance at a national research meeting. DESIGN AND SETTING: All abstracts submitted to the American Heart Association's annual Scientific Sessions research meeting from 2000-2004. Abstract review included the author's name and institution (open review) from 2000-2001, and this information was concealed (blinded review) from 2002-2004. Abstracts were categorized by country, primary language, institution prestige, author sex, and government and industry status. MAIN OUTCOME MEASURE: Likelihood of abstract acceptance during open and blinded review, by abstract characteristics. RESULTS: The mean number of abstracts submitted each year for evaluation was 13,455 and 28.5% were accepted. During open review, 40.8% of US and 22.6% of non-US abstracts were accepted (relative risk [RR], 1.81; 95% confidence interval [CI], 1.75-1.88), whereas during blinded review, 33.4% of US and 23.7% of non-US abstracts were accepted (RR, 1.41; 95% CI, 1.37-1.45; P<.001 for comparison between peer review periods). Among non-US abstracts, during open review, 31.1% from English- speaking countries and 20.9% from non-English-speaking countries were accepted (RR, 1.49; 95% CI, 1.39-1.59), whereas during blinded review, 28.8% and 22.8% of abstracts were accepted, respectively (RR, 1.26; 95% CI, 1.19-1.34; P<.001). Among abstracts from US academic institutions, during open review, 51.3% from highly prestigious and 32.6% from nonprestigious institutions were accepted (RR, 1.57; 95% CI, 1.48-1.67), whereas during blinded review, 38.8% and 29.0% of abstracts were accepted, respectively (RR, 1.34; 95% CI, 1.26-1.41; P<.001). CONCLUSIONS: This study provides evidence of bias in the open review of abstracts, favoring authors from the United States, English-speaking countries outside the United States, and prestigious academic institutions. Moreover, blinded review at least partially reduced reviewer bias.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.003 | 0.001 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.001 | 0.002 |
| Insufficient payload (model declined to judge) | 0.006 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".