Using a commitment‐to‐change strategy to assess faculty development
Bibliographic record
Abstract
‘Cabin Fever’ is an established University of Calgary faculty development programme offered annually to help rural educators improve their teaching and assessment skills. Teachers participate in five 80-minute workshops (out of 22) and one plenary session and interact informally with colleagues during lunches and dinners. Workshop topics have included: teaching procedural skills; teaching professionalism; teaching empathy; teaching multiple levels of learners; the resident as teacher; assessment and feedback; learning technologies; patient safety, and building effective workplaces. Post-course evaluations indicated the course was successful. There was no information about the programme’s ongoing impact on teaching staff and therefore potentially on learners. Data were required to defend the programme’s continuance after 10 years because it requires significant resources. We adopted ‘commitment-to-change statements’ (CTCs) to enhance course evaluation. At the end of the programme, doctors were asked to identify three changes they planned to make in the next 3 months and their level of commitment to making each change (using a scale of 1–4, where 1 = low and 4 = high). Three months later, participants were asked whether the intended changes were complete, still in progress, incomplete or would not be undertaken. We analysed the quantitative data descriptively and themed the commitments. Approximately 40% (35/81) of the participants completed CTCs at the end of the programme. Over 75% (n = 27) provided complete data for 79 changes (2.9 per doctor) 3 months after the course. Initial commitment to making the changes was high at 3.36/4. At 3 months, 33 (41.8%) of the changes had been implemented, 24 (30.4%) had been partially implemented, 21 (26.6%) could not be implemented, and one (1.3%) had been abandoned. Making changes in approaches to teaching was identified in 31 CTCs. These included improving communication skills (by spending more time on listening), enabling residents to teach students, and adopting new competency-based assessment tools and new approaches to teaching procedural skills. A total of 19 CTCs related to assessment and feedback (e.g. adopting feedback frameworks taught, giving specific or regular feedback, planning feedback). There were 15 commitments to improve the use of information technology (IT) and electronic resources (e.g. setting up a website for practice, using websites, creating lists of websites for resident or patient teaching). There were nine commitments to improve role-modelling, most of which related to demonstrating the benefits of family medicine as a career. The final five commitments focused on office management for effectiveness and error disclosure. The CTCs that the doctors were able to implement and carry out themselves (e.g. improving communication or observation of trainees) were likely to have been adopted fully or partially. By contrast, many of the 22 changes the doctors had been unable to implement or had abandoned were more complex and required additional time or the opportunity to implement. Almost half (7/15) of the plans for IT and electronic resources were incomplete or had been abandoned, whereas only a third (12/31) of plans relating to approaches to teaching had not succeeded. The CTC process provided new data with which to assess our programme because the CTCs were directly associated with the content provided. Changes were in the directions intended. Encouraging all attendees to complete their CTC forms is critical to understanding programme impact.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.028 | 0.073 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.002 |
| Bibliometrics | 0.007 | 0.004 |
| Science and technology studies | 0.002 | 0.003 |
| Scholarly communication | 0.003 | 0.003 |
| Open science | 0.002 | 0.006 |
| Research integrity | 0.002 | 0.003 |
| Insufficient payload (model declined to judge) | 0.004 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".