Developing Competency-assessment Tools to Measure the Family Physicianʼs Ability to Respond to the Needs of the Community

There has been an increase in the emphasis placed on the importance of primary care and the way primary care services are delivered worldwide. Systems for delivery of primary health care must provide reliable, timely, comprehensive care to defined groups of people based upon their needs.1 The Institute of Medicine defines primary care as “The provision of integrated, accessible health care services by clinicians who are accountable for addressing a large majority of personal health care needs, developing a sustained partnership with patients, and practicing in the context of family and community.”2 In Canada, two of the four principles upon which family physicians should model their practices are (1) family medicine is community-based, and (2) a family physician is a resource to a defined practice population.3 Despite the recognition that community should play a larger role in the work of physicians, there is little agreement on how this should be done4 and, thus, how it should be taught. Medical schools have a unique purpose of selecting and educating competent, caring physicians capable of meeting society's expectations.5 They must accept some responsibility for the health of defined communities related to health care reform.6 Family medicine training programs must ensure that residents learn to recognize community-oriented issues, including the needs of the communities, and must develop ways to measure residents' competence. Examination methods can greatly influence the approach to teaching and learning within the medical profession.7 Thus, one of the driving forces that can help change the training of family physicians is the development of community-oriented competency-assessment tools. The Residency Program Accreditation and Certification Manual from the College of Family Physicians of Canada1 states: “Residents in family medicine must learn how to identify and respond effectively to the needs of communities.” This learning objective was tested in this study, recognizing that doctor—patient interactions provide family physicians opportunities to identify and respond to health issues that are reflective of a larger community. In this pilot study, community was defined as any group, population, or cluster of people sharing at least one common characteristic such as geographic location, illness, ethnicity, or housing condition.8 For example, in the case of a child with new-onset peanut allergy, the role of the family physician was to ensure that the mother informed the school community of the child's peanut allergy and that she investigated the school's policies for prevention and treatment of anaphylaxis. In all cases within the present study, clearly defined communities were identified, and the family physicians were given the opportunity to respond to health issues that were reflective of a larger community. The purpose of the pilot study was to develop valid and reliable evaluation tools to measure community-responsive competencies of family physicians. The specific objectives of the study were: to test an operational definition of community responsiveness that recognizes family physicians have a unique opportunity to identify and respond to the needs of a defined community based upon the identification of community-oriented issues found during doctor—patient interactions; to pilot test various evaluation tools to measure the community-oriented practice of family medicine; and to determine the characteristics of each evaluation tool relative to validity, reliability, and reproducibility. Method Subjects. Thirty-five participants were recruited. Ten fourth-year medical students, nine first-year family medicine residents (PGY1s), and seven second-year family medicine residents (PGY2s) voluntarily participated in the study after receiving an e-mail invitation. Canadian family medicine residency programs provide two years of training for their residents. “Community-responsive” family physicians were selected by asking key informants to identify family physicians who were known to assimilate into the community and coordinate resources for their patients in an exemplary manner, and who were aware of and responsive to the sociocultural needs of their patients. These selection criteria were derived from the work of Pathman et al., who described the dimensions of what American primary care physicians do in the community.4 Ninety-three Toronto community agencies providing a range of services for populations, including those related to age, gender, ethnicity, and socio-economic status, were contacted by e-mail. In addition, seven faculty members of the University of Toronto's Department of Family and Community Medicine were contacted as key informants. Forty names of community-responsive family physicians and general practitioners were generated. All 40 were contacted, and nine physicians volunteered to participate, based on their interest and availability for the study. Six men and three women physicians, with experience ranging from three to 30 years in practice, participated. Competency assessment tools. The three assessment tools developed and tested in this study were structured to test similar content areas. The core community-responsive competencies (including knowledge, skills, and attitudes) were derived from a review of the literature and based on the accreditation guidelines for residency programs related to the principle “Family Medicine is Community Based,” as developed by the College of Family Physicians of Canada. For each tool, candidates were asked to manage a problem that reflected a larger community-oriented health issue derived from the doctor—patient interaction. The community-responsive competencies tested included the candidate's abilities to demonstrate knowledge of community resources and the roles of allied health professionals, to consult medical specialists appropriately, and to coordinate care for the patient. Last, all tools were designed to capture the candidates' willingness to facilitate or intervene, where appropriate, relative to health problems that may affect a defined community. The tools tested both community-responsive knowledge and skills and biopsychosocial knowledge and skills for the purpose of establishing divergent validity of the community-responsive measures. A specific case within the study involved a young man with a recent diagnosis of a seizure. The competent community-responsive physician would recognize that this man's seizure could affect the lives of the patient and others in the community if the patient continued to drive. The candidate was expected to manage the seizure diagnosis, consult specialists appropriately, demonstrate knowledge of community resources that might be helpful to him, and report his diagnosis to the local governing body on transportation, thus ensuring that the safety of the patient and the community is upheld. Three assessment tools were developed and tested: Simulated oral examinations (SOOs). Standardized patients were used to test the candidates' skills in developing diagnoses and management plans related to three specific case scenarios. For each patient encounter, examiners marked a series of checklist items on a binary scale of “done” or “not done.” The checklist items were clustered into a series of categories or dimensions of performance. Based on the pattern of checklist items marked “done,” each dimension was scored as either “pass” or “fail.” The proportion of dimensions passed was calculated as the final score for each SOO case. Both the standardized patient and a physician examiner observing the case scored the candidates. SOOs have been shown previously to be useful in measuring doctor—patient communication skills.9 Short-answer management problems (SAMPs). The SAMPs consisted of three written clinical scenarios, each followed by a series of questions requiring short answers. Scoring was based on a series of checklist items marked on a binary scale of “done” and “not done.” The final score was calculated by determining the proportion of items marked “done” on each SAMP case. Two examiners marked each SAMP independently. SAMPs measure factual knowledge, recall, and problem-solving skills.10 Chart-stimulated recall (CSR). Candidates were asked to bring in patients' charts or written summaries of four cases of patients with whom they had had direct clinical involvement in a manner reflective of family medicine. The cases were used to generate discussion of the candidates' approaches to the patients' diagnoses and clinical management. A five-point rating scale, adapted from a study by Norman and Davis,11 was developed to address patient-centered, family-focused, and community-responsive dimensions of clinical care. The anchors were constructed based on Stewart's article on scoring of patient-centeredness,12 and Pathman's study4 on the dimensions of what physicians do in the community. All of the CSR anchors involved extensive behavioral descriptors at the first, third, and fifth markers. A cumulative score was calculated from the summation of all scores from each dimension tested. Two CSR cases were used and each was audiotaped. A physician—examiner using the rating scale marked each candidate, while another physician—examiner marked each candidate based on the transcriptions of the audiotaped interactions. CSR measures a candidate's performance and provides information about how the candidate makes decisions about diagnosis, investigations, and treatment, as well as issues related to the environment, the system, and characteristics of the health care provider.15 CSR has been found to have good validity and reliability.15–17 Three cases for each of the SOOs and SAMPs and two cases for the CSR were tested. Each case was designed as a 15-minute station, and one 15-minute break was provided, for a total of nine stations. With two minutes between stations, the total testing time was approximately two and a half hours. After the candidates completed the stations, they were given an attitude questionnaire and invited to participate in a focus group session to provide feedback on the assessment tools. The candidates were asked to complete a brief survey measuring their attitudes toward responding to the needs of the community and the community-oriented principles of family medicine. The attitude questionnaire given to the participants was developed to measure both the significance of carrying out community-responsive activities within their medical careers and competence in carrying out these tasks within their practices. Participants were asked to rate their responses on a five-point scale (1 = being insignificant/incompetent and 5 = very significant/very competent). The attitude questionnaire was developed using a combination of questions related to community-oriented care by family physicians from surveys used by Pathman4 and Oandasan.18 Two iterations of the study protocol were implemented, one in the morning and the second in the afternoon. Each iteration, including the attitude questionnaire and the focus group, lasted four hours and 15 minutes. Analysis. The community-responsive measures for each of the competency-assessment tools were assessed for reliability and validity. Inter-rater reliability for each tool was assessed using Pearson product—moment correlation coefficients. There were two independent raters for the SOOs (physician—examiner and simulated patient), SAMPs (two physician markers), and CSR (physician—examiner and transcript examiner). Internal consistency of the multiple stations for each tool was assessed using Cronbach's alpha. Construct validity was assessed by comparing the community-responsive scores for the medical students, PGY1 family medicine residents, PGY2 family medicine residents, and family physicians using one-way analysis of variance (ANOVA) and calculating both the statistical significance and the proportion of variance accounted for by level of experience. Divergent validity was assessed for each tool by comparing candidates' community-responsive scores on the tool with their scores on the “other” dimensions of ability that were assessed by the tool (such as biomedical knowledge or the management of biopsychosocial issues). High correlations would be taken as cause for doubting the ability of the tools to assess community responsiveness independent of these other constructs. Convergent validity was assessed through a Pearson correlation of the community-responsiveness scores generated for each of the three measures. Results The reliability and validity of the competency-assessment tools may be found in Table 1. Inter-rater reliability based upon two independent scorers for each tool was found to be high for all three tools. Both the SOOs and the SAMPs had low inter-station reliability. In contrast, the CSR was found to have high inter-station reliability. This finding was interesting, given that only two cases were tested using CSR. Spearman Brown prophecy reveals that with four stations, a reliability of .84 could have been generated for one hour of testing time.TABLE 1: Reliability and Validity of Three Competency Assessment Tools Administered to Four Groups, University of Toronto, 2000Construct validity was low for the SAMPs and moderate for the SOOs but was quite high for CSR, with substantially higher scores by the “community-responsive family physicians” versus the PGY1s, PGY2s, and fourth-year medical students, accounting for 57% of the variance in candidate scores. The divergent validity was calculated based on the correlation between candidates' community scores on each tool and their scores on the “other” constructs (such as biomedical knowledge) measured by the tool. The correlations were found to be moderate to high for all tools, raising some concern regarding the measurement of community responsiveness in this study, especially for the CSR tool. Concurrent validity findings were disappointing. The correlations found between CSR and the SAMP (r = .13), CSR and the SOO (r = .35), and SAMP and SOO (r = .10) were all low, probably because of the poor internal-consistency findings of the assessment tools. The findings of the attitude questionnaire reveal that there was no significant difference between the scores of the participants. The scores indicate that the questionnaire, originally developed to determine attitudes toward community responsiveness and competence within this domain, was unable to discriminate between participants. The mean for significance among all participants was .858 and that for competence was .700. Discussion A family physician who is capable of responding to the needs of the community is a key element in the future of primary health care. It is timely, therefore, to identify competency-assessment tools to measure a family physician's knowledge, skills, and attitude toward the principles of family medicine that are community oriented. This study was developed to begin to develop valid and reliable competency-assessment tools using a specific operational definition of community responsiveness. This project was successful in meeting its three specific objectives. First, the operational definition of community responsiveness used in this study, recognizing that doctor—patient interactions provide family physicians with the opportunity to identify and respond to health issues that are reflective of a larger community, was measurable. Second, evaluation tools were successfully tested, and third, the psychometric properties of the tools were identified. The CSR is the tool that shows the most promise. With only two cases tested, its high inter-rater reliability, good inter-station reliability, and high construct validity suggest that it may be implemented as a summative assessment if larger numbers of cases are used. Although divergent validity was found to be relatively high among all tools, this finding was not unexpected for the CSR. The CSR was designed to measure patient-centered, family-focused, and community-responsive approaches to the management of a patient's care. These specific parameters may be co-related in a holistic manner and thus unlike biomedical approaches to clinical management that the other tools (SAMPs and SOOs) specifically measured in their cases. The relatively high divergent validity in the SAMPs and SOOs raises questions about the measurement of community responsiveness as a domain and perhaps the need for better ways to isolate its properties to distinguish it further from biomedical management. Although the CSR was developed as a summative assessment tool, the feedback received from both participants and examiners indicates that CSR could also be used as a formative assessment tool. Many of the trainees, in particular, found it to be very useful in stimulating their depth of understanding of their clinical cases presented through the questions posed by the examiners. It is believed that this tool, which measures patient-centered, family-focused, and community-responsive approaches to clinical management, is the first of its kind to be used in the family medicine setting. More research with CSR in the future will determine its true value. There were two major limitations to this study. First, as this was a pilot study, a small sample size was used. This may have affected the results found for the SAMPs, the SOOs, and the Attitude Questionnaire, but CSR showed promise. Second, it was an initial attempt to measure an important but not easily defined measure of competence in community responsiveness. An operational definition was used, but it may not have been a comprehensive definition that reflects how family physicians respond to the needs of communities. Thus, this study's first attempt will provide groundwork in this area, which is sure to grow, given the nature of the increasing importance of primary health care and primary health care reform. The World Health Organization describes the “Five Star Doctor” for the 21st century as one who is community-oriented, reconciling both individual and community health requirements and initiating actions on behalf of the community.19 The time to teach this quality to medical trainees, particularly family medicine trainees, is now. If assessment drives curricular change and if curricular change is needed, then more studies are required to encourage the development of better assessment tools to measure ways in which family physicians respond to the needs of community. This pilot study hopes to contribute to the bank of community-oriented assessment resources that are greatly needed at this time.

Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.