Measuring competence in healthcare learners and healthcare professionals by comparing self-assessment with objective structured clinical examinations: a systematic review
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
Executive summary Background The measure of clinical competence is an important aspect in the education of healthcare professionals. Two methods of assessment are typically described; an objective structured clinical examination and self-assessment. Objectives To compare the accuracy of self-assessed competence of healthcare learners and healthcare professionals with the assessment of competence using an objective structured clinical examination. Inclusion criteria Types of participants All healthcare learners and healthcare professionals including physicians, nurses, dentists, occupational therapists, physiotherapists, social workers and respiratory therapists. Types of intervention Studies in which participants were first administered a self-assessment (related to competence), followed by an objective structured clinical examination; the results of which were then compared. Types of outcomes Competence, confidence, performance, self-efficacy, knowledge and empathy. Types of studies Randomized controlled trials, non-randomized controlled trials, controlled before and after studies, cohort, case control studies and descriptive studies. Search strategy A three-step search strategy was utilized to locate both published and unpublished studies. Databases searched were: Medline, CINAHL, Embase, ERIC, Education Research Complete, Education Full Text, CBCA Education, GlobalHealth, Sociological Abstracts, Cochrane, PsycInfo, Mosby's Nursing Consult and Google Scholar. No date limit was used. Methodological quality Full papers were assessed for methodological quality by two reviewers working independently using the Joanna Briggs Institute Meta-Analysis of Statistics Assessment and Review Instrument (JBI-MAStARI). Data collection Details of each study included in the review were extracted independently by two reviewers using an adaptation of the standardized data extraction tool from JBI-MAStARI. Data synthesis Meta-analysis was not possible due to methodological and statistical heterogeneity of the included studies. Hence study findings are presented in narrative form. The data was also analyzed using 'The Four Stages of Learning' model by Noel Burch. Results The search strategy located a total of 2831 citations and 18 studies were included in the final review. No articles were removed based on the critical appraisal process. For both competence and confidence, the majority of studies did not support a positive relationship between self-assessed performance and performance on an OSCE. Conclusions Study participants' self-assessed competence or confidence was not confirmed by performance on an objective structured clinical examination. An accurate self-assessment may be threatened by over confidence and high performers tend to underestimate their ability. It is theorized that this disparity may in part be due to the stage that the learner or professional is in, with regard to knowledge and skill acquisition. Educators need to examine their evaluation methods to ensure that they are offering a varied and valid approach to assessment and evaluation. Notably, if self-assessment is to be used within programs, then learners need to be taught how to perform consistent and accurate self-assessments. Implications for practice It is important that educators understand the limitations within the evaluation of competence. Key aspects are the recognition of the stage that the learner is in with regard to skill acquisition and equipping both learners and professionals with the ability to perform consistent and accurate self-assessments. Implications for research There is a need for standardization on how outcomes are identified and measured in the area of competence. Further, identifying the leveling of an OSCE and the appropriate number of stations is required.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.021 | 0.002 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.007 | 0.000 |
| Bibliometrics | 0.000 | 0.001 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it