EQual, a Novel Rubric to Evaluate Entrustable Professional Activities for Quality and Structure
Bibliographic record
Abstract
PURPOSE: Entrustable professional activities (EPAs) have become a cornerstone of assessment in competency-based medical education (CBME). Increasingly, EPAs are being adopted that do not conform to EPA standards. This study aimed to develop and validate a scoring rubric to evaluate EPAs for alignment with their purpose, and to identify substandard EPAs. METHOD: The EQual rubric was developed and revised by a team of education scholars with expertise in EPAs. It was then applied by four residency program directors/CBME leads (PDs) and four nonclinician support staff to 31 stage-specific EPAs developed for internal medicine in the Royal College of Physicians and Surgeons of Canada's Competency by Design framework. Results were analyzed using a generalizability study to evaluate overall reliability, with the EPAs as the object of measurement. Item-level analysis was performed to determine reliability and discrimination value for each item. Scores from the PDs were also compared with decisions about revisions made independently by the education scholars group. RESULTS: The EQual rubric demonstrated high reliability in the G-study with a phi-coefficient of 0.84 when applied by the PDs, and moderate reliability when applied by the support staff at 0.67. Item-level analysis identified three items that performed poorly with low item discrimination and low interrater reliability indices. Scores from support staff only moderately correlated with PDs. Using the preestablished cut score, PDs identified 9 of 10 EPAs deemed to require major revision. CONCLUSIONS: EQual rubric scores reliably measured alignment of EPAs with literature-described standards. Further, its application accurately identified EPAs requiring major revisions.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.017 | 0.065 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.002 |
| Bibliometrics | 0.010 | 0.003 |
| Science and technology studies | 0.001 | 0.002 |
| Scholarly communication | 0.002 | 0.003 |
| Open science | 0.002 | 0.004 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.004 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".