The Interrater Reliability of a Validated Bronchiolitis Severity Assessment Tool
Bibliographic record
Abstract
BACKGROUND: We previously constructed and tested a bronchiolitis severity assessment tool in 2 independent hospitals. The model uses age, work of breathing, dehydration and tachycardia to successfully predict disease severity. OBJECTIVE: To prospectively measure the interrater reliability of a bronchiolitis severity assessment tool and of its component variables. DESIGN: Prospective observational survey. SETTING: A county teaching hospital emergency department serving a mixed urban and rural population with an emergency medicine residency program in 2-3-4 format. SUBJECTS: Thirty-two physicians evaluated a convenience sample of children aged less than 18 months presenting to the emergency department with a clinical diagnosis of bronchiolitis during a single season. METHODS: Two physicians independently examined each patient. Each physician completed a physical examination template that included the variables used in the severity assessment tool. Interrater agreement was measured for the variables work of breathing and dehydration and for the tool as a whole using a weighted kappa statistic. RESULTS: One hundred and forty-six cases were enrolled. Twenty-five were dropped for incomplete data collection. The actual weighted agreement on overall classification was 92%; expected, 73%, kappa = 0.676; P < 0.0001. The actual weighted agreement for dehydration was at 95%; expected, 92%, kappa = 0.305; P = 0.0001. The agreement for work of breathing was 95%; expected, 86%; kappa = 0.611; P < 0.0001. The overall model showed better interrater reliability than its individual components. CONCLUSIONS: Overall interrater reliability for this bronchiolitis severity assessment tool is substantial.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.000 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.001 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.001 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".