Designing a User-Friendly Context-Specific Assessment Tool for Community-Based Teachers
Notice bibliographique
Résumé
Graduate medical education (GME) programs must ensure that they are able to collect accurate information about resident competence through assessment tools that are fit for purpose. An assessment form or process is called “fit for purpose” when there is good alignment between the tool (how something is assessed) and the intent (the specific knowledge, skill, or attitude/belief that is being assessed).In our family medicine GME program, we identified that the generic workplace-based summative assessment tool provided to community-based family medicine obstetrics (FMOB) clinical teachers was not fit for purpose. As a result, these teachers were uncertain about program expectations regarding competence, the assessment form was challenging and frustrating to complete, and our program struggled to extract useful assessment data from completed assessment forms. To address this issue, we took a systematic approach to develop a workplace-based assessment tool that was specific to the clinical context of FMOB and user-friendly for clinical teachers.Our goals were to develop: (1) a fit-for-purpose workplace-based assessment tool for community-based FMOB teachers that could be used for accurate assessment without the need for faculty development, and (2) an evidence-guided process for designing similar context-specific tools in the future. We met our first goal through designing the FMOB competence rubric (FMOB-CR) tool. The FMOB-CR is comprised of 2 elements. The first is a rubric containing specific statements and examples of resident performance. The rubric clearly outlines the expected level of competence on a 3-point rating scale (“cause for concern,” “acceptable competence,” “exemplary competence”), organized by the 6 Skill Dimensions of Family Medicine (similar to CanMEDS roles).1 The second element is a simple online assessment form in which FMOB teachers specify their resident's level of competence in each Skill Dimension, guided by the statements and examples in the rubric.The FMOB-CR uses plain language and context-specific clinical examples along with a simple online assessment form to clearly communicate expectations of competence to teachers (and residents) without additional faculty development. This user-friendly tool should make it easier for FMOB clinical teachers to better identify residents who are underperforming, which will allow the program to be more effective in intervening and supporting residents in difficulty.To meet our second goal, we developed an evidence-guided process for designing workplace-based assessment tools that are fit for purpose for specific clinical contexts. Our design process included: assembling a tool development team with relevant expertise, including a resident; consultations with FMOB teachers and residents; environmental scan and review of local assessment forms and assessments used by other FMOB GME programs; a modified Delphi process with local GME program faculty to develop and refine the table of competence statements; a consensus-building process within the research team to revise the statements for the rating scale; and implementation of the assessment rubric with concurrent collection of validity evidence. For both the FMOB-CR and the development process, we collected validity evidence according to Messick's unified concept of validity.2The Table details the validity evidence collected to date. Fifteen FMOB teachers surveyed showed strong agreement across 5 items about the utility of the rubric in allowing them to accurately assess residents (overall M=3.25/4, Likert scale 1=strongly disagree to 4=strongly agree), and across 3 items about the usefulness of the rubrics in helping them to understand program expectations of resident competence (overall M=3.36/4); 14 of 15 teachers preferred the new form to the old one.We hope that the worked example of the FMOB-CR and its development process may serve as a blueprint for other institutions to develop context-specific assessment tools.
Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.
Comment cette classification a été obtenuedéplier
Prédiction distillée sur la base complète
Imitation des enseignantsNi prévalence calibrée, ni vérité terrain. Validation humaine à venir. Apprise à partir de 10 348 étiquettes directes de Codex et de 10 348 étiquettes directes de Gemma. Le mode candidate est l'union des têtes enseignantes seuillées; le consensus est leur intersection. Ces sorties portent le statut machine_predicted_unvalidated et ne sont ni des étiquettes humaines ni des étiquettes directes de modèles de pointe.
Scores Codex et Gemma par catégorie
| Catégorie | Codex | Gemma |
|---|---|---|
| Métarecherche | 0,007 | 0,009 |
| Méta-épidémiologie (sens strict) | 0,000 | 0,000 |
| Méta-épidémiologie (sens large) | 0,000 | 0,000 |
| Bibliométrie | 0,001 | 0,001 |
| Études des sciences et des technologies | 0,000 | 0,000 |
| Communication savante | 0,000 | 0,000 |
| Science ouverte | 0,000 | 0,000 |
| Intégrité de la recherche | 0,000 | 0,001 |
| Charge utile insuffisante (le modèle a refusé de juger) | 0,000 | 0,000 |
Scores machine (provisoires)
Les deux têtes enseignantes du modèle étudiant, lues sur ce travail. Un score ordonne la base pour la relecture; il n'affirme jamais une catégorie, et le statut de validation accompagne chaque rangée tel quel.
Scores de référence d'un modèle non mature (critères de maturité non atteints, 7 itérations). Un score ordonne; il n'affirme jamais une catégorie.
score_only:v0-immature-baseline · tel quel depuis la passe de notation : score_only signifie que le nombre peut ordonner les travaux, et qu'aucune étiquette de catégorie n'en découleClassification
machine, non validéePrédiction automatique; un appel candidat d’une seule tête enseignante, pas un consensus.
Le détail, modèle par modèle et score par score, se trouve en fin de page sous « Comment cette classification a été obtenue ».