How Good Is Good? Students and Assessors’ Perceptions of Qualitative Markers of Performance
Bibliographic record
Abstract
BACKGROUND: Qualitative markers of performance are routinely used for medical student assessment, though the extent to which such markers can be readily translated to actionable pieces of information remains uncertain. PURPOSE: To explore (a) the perceived value to be indicated by descriptor phrases commonly used for describing student performance, (b) the perceived weight of the different performance domains (e.g. communication skills, work ethic, knowledge base, etc), and (c) whether or not the perceived value of the descriptors changes as a function of the performance domains. METHODS: Five domains of performance were identified from the thematic coding of past medical student transcripts (N = 156). From the transcripts, 91 distinct descriptors indicating the language commonly used by assessors were also identified. From the list of 91 descriptors, Thurstone's method of equal-appearing intervals was used to extract 10 descriptors that were representative of the continuum of student performance. A modified paired comparisons method was then used to enable the relative ranking of each of 10 descriptors combined with each of 5 different domains of performance. A web-based survey was used to collect responses from participants (N = 209), which consisted of medical students and faculty members who were previously involved in student assessment. RESULTS: Results demonstrated that respondents did not simply sum positive and negative descriptors in a uniform manner. Rather, comments on some domains (e.g., "ability to apply patient centred medicine") were seen as particularly positive when associated with positive descriptors but not particularly negative when associated with negative descriptors. For others (e.g., "receptivity and responsiveness to feedback") the reverse was true. Comments on "knowledge-base" elicited a relatively muted perception at both ends of the scale. Finally, the results also revealed moderate misalignment in the perceptions of assessors and students. CONCLUSIONS: The findings from this study suggest that the use of any given descriptor conveys slightly different meaning dependent on the context in which it is used. This helps to address some key issues surrounding the application of qualitative markers to performance assessment in medical education.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.003 | 0.003 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".