Interval estimation for a proportion using a double-sampling scheme with two fallible classifiers
Bibliographic record
Abstract
Double-sampling schemes using one classifier assessing the whole sample and another classifier assessing a subset of the sample have been introduced for reducing classification errors when an infallible or gold standard classifier is unavailable or impractical. Inference procedures have previously been proposed for situations where an infallible classifier is available for validating a subset of the sample that has already been classified by a fallible classifier. Here, we consider the case where both classifiers are fallible, proposing and evaluating several confidence interval procedures for a proportion under two models, distinguished by the assumption regarding ascertainment of two classifiers. Simulation results suggest that the modified Wald-based confidence interval, Score-based confidence interval, two Bayesian credible intervals, and the percentile Bootstrap confidence interval performed reasonably well even for small binomial proportions and small validated sample under the model with the conditional independent assumption, and the confidence interval derived from the Wald test with nuisance parameters appropriately evaluated, likelihood ratio-based confidence interval, Score-based confidence interval, and the percentile Bootstrap confidence interval performed satisfactory in terms of coverage under the model without the conditional independent assumption. Moreover, confidence intervals based on log- and logit-transformations also performed well when the binomial proportion and the ratio of the validated sample are not very small under two models. Two examples were used to illustrate the procedures.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.075 | 0.257 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.003 | 0.002 |
| Bibliometrics | 0.004 | 0.003 |
| Science and technology studies | 0.001 | 0.003 |
| Scholarly communication | 0.002 | 0.004 |
| Open science | 0.003 | 0.003 |
| Research integrity | 0.003 | 0.003 |
| Insufficient payload (model declined to judge) | 0.003 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".