A comparison of unbalanced rating scales in scoring competency assessments
Bibliographic record
Abstract
Assessments of professional competencies often rely on the judgement of peer assessors who observe performance and examine professional artefacts. The assessors' judgements may, be influenced, however, by the rating scales they use to record their judgements. Much research has examined the characteristics of rating scales for appraisals of professional competence for hiring or compensation decisions; less research has focused on regulation-driven competency assessments intended to identify professionals who do not meet the standards for their profession and to assign a course of remediation specific to the level of deficiency. The College of Physiotherapists of Ontario's (CPO's) 2004 pilot test of its redesigned competency assessment program provided a unique opportunity to investigate the effects of rating scale characteristics in a regulatory context. Most registrants consistently meet competency standards; however, for those who do not, fine distinctions in deficiency are essential for decision-making about the appropriate remediation strategy. It was expected that a rating scale that was unbalanced in the direction of negative ratings and had more rating points would have greater discriminatory power and therefore greater utility for the CPO. Two unbalanced rating scales were developed for the study: negatively-labelled 3-point and 4-point scales. The results showed that, regardless of the scale used, reliability was near perfect when the rating for a competency area was Meets standards. For competency areas that did not meet standards, ratings varied considerably. This may have been due to: (1) varying levels of leniency and stringency among assessors, (2) uncertainty on the part of assessors about the degree of deficiency that was to correspond to the labels for sub-standard competency, or (3) insufficient information reported by on-site assessors in the assessment reports to enable off-site assessors to fully appreciate the practice situation and competency levels. Assessors expressed preferences for more points on their assigned scale to accommodate a rating of excellence and rewording the negative labels with positive terms to indicate degree of improvement required rather than degree of deficiency.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.147 | 0.419 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.002 |
| Bibliometrics | 0.006 | 0.004 |
| Science and technology studies | 0.001 | 0.002 |
| Scholarly communication | 0.004 | 0.005 |
| Open science | 0.002 | 0.006 |
| Research integrity | 0.001 | 0.002 |
| Insufficient payload (model declined to judge) | 0.005 | 0.002 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".