The Reliability and Validity of a Paediatric Script Concordance Test with Medical Students, Paediatric Residents and Experienced Paediatricians
Bibliographic record
Abstract
Background: The Script Concordance (SC) approach was used as an alternative test format to measure the presence of knowledge organization reflective in one’s clinical reasoning skills (i.e., diagnostic, investigation and treatment knowledge).Methods: The present study investigated the reliability and validity of a 40-item paediatric version of the SC test with three groups representing 53 medical students (novices), 42 paediatric residents (intermediates) and 11 paediatricians (experts).Results: A comparison between scoring techniques based on experts’ ratings of the items showed internal reliability coefficients from .74 for the one-best answer up to .78 for alternative scoring techniques. An ANOVA showed an increase in test performance from medical students through to expert paediatricians (F(2,103) = 84.05, p < .001), but did not differentiate between the postgraduate year 1 to 3 paediatric residents. A large effect size (Cohen’s d) difference of 1.06 was found between medical students and residents total SC test scores.Conclusions: These results support other findings indicating the SC test format can be used to differentiate between the clinical reasoning skills of novices, intermediates and experts in paediatrics. An alternative scoring method that includes one best answer and partial marks was also supported for grading SC test items.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.004 | 0.546 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.001 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.002 |
| Insufficient payload (model declined to judge) | 0.001 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".