Revisiting ‘The effect of assessments and examinations on the learning of medical students’

Editor's note: As part of our 50th volume celebrations, Medical Education is looking back at its most impactful articles, as defined by citation count. The most cited articles from each 5-year interval were identified and the original authors of one of them (or other knowledgeable scholars if the original authors could not be found) were asked to comment on the state of the field at the time of publication, the impact of the article, and what we have learned since then. The article illustrated in Figure 1 was one of the most cited articles in our journal in the 1982–1986 period. To see the other top-cited articles from Volumes 1–50 please view the interactive PDF by visiting www.mededuc.com. ‘Assessment drives learning’ has become a much quoted aphorism. Yet how many educational institutions, regulatory bodies or teachers appreciate the magnitude of the impact of their assessments on the learning of their students? Failure to do so may have significant unintended effects on students’ learning behaviour, medical knowledge and clinical competence. The paper on this topic published by Kerry Jaeger and myself nearly 35 years ago,1 one of those now being celebrated in this issue (Fig.1), described the effects of changes to the assessment methods introduced as part of a curriculum revision that undermined the intended outcomes. The main intent of the final year of the new course was to reduce didactic knowledge-based teaching to a minimum in order to maximise the time spent on the wards and thereby to prepare students more effectively for their roles as interns. However, during the second year of the new programme, students were observed to be expending more of their effort on acquiring theoretical knowledge than on obtaining clinical skills on the wards. A survey of these students revealed that this was the consequence of changes to the revised assessment scheme. The traditional final-year clinical viva voce had been replaced (for good psychometric reasons – see later) by ward ratings. However, the previous end-of-year multiple-choice question (MCQ) examination had been retained. After only one cycle of the revised final year, the students had worked out that the chances of failing as a result of poor ward ratings were minimal and that success or failure were in reality almost entirely based on performance in the MCQ test. The faculty's remedial strategy was not to change the curriculum or the current assessments, but to reintroduce an end-of-year clinical examination to counterbalance the dominating effect of the MCQ test. Our study1 was conducted to monitor the impact of this strategy. How many educational institutions appreciate the magnitude of the impact of assessments on the learning of their students? Many current readers of this journal may not appreciate how different the medical education landscape was around 35 years ago when this work was conducted and hence it may be of some interest to reflect on the context in which this study took place. At this time, most medical schools in English-speaking countries still ran so-called traditional curricula, split into pre-clinical and clinical components, both of which were strictly discipline-based. Worldwide there were only a handful of medical education departments to initiate and guide innovation and development. Improvements in assessment had rarely advanced beyond the introduction of MCQs as an alternative to essay and short-answer format-based methods. However, things were about to change. In 1975 I was privileged to undertake a sabbatical at the University of Southern California (USC) with Professor Stephen Abrahamson, head of one of very few internationally recognised departments of medical education. These were momentous times in medical education and USC was at the forefront of innovation. Abrahamson and his colleagues had developed Sim 1 and Sim 2, the first computer-controlled manikins,2 and Howard Barrows had pioneered simulated patients.3 Elsewhere in the USA, people were conducting experiments using various paper- and computer-based patient management problems.4 In Dundee, Ronald Harden and colleagues had just published their landmark paper on the objective structured clinical examination (OSCE).5 To round this off, McMaster University in Canada was triggering a revolution in medical teaching by challenging the whole structure of the traditional curriculum with its problem-based learning (PBL) approach.6 Many current readers of this journal may not appreciate how different the medical education landscape was 35 years ago While at USC I had the opportunity to do some part-time work as a clinical teacher in the Los Angeles County Hospital. It soon became clear that I was unique in taking my students onto the wards to conduct my teaching around the bedside. In the usual approach the majority of teaching was conducted in tutorial rooms and focused on theory more than it did on clinical skills. By this time, I had become aware of the research of J P Hubbard and colleagues, which had shown that the traditional clinical examinations conducted by the National Board of Medical Examiners (NBME), which involved a bedside oral test, were so unreliable that they had been abandoned in the 1970s.7 This made me speculate that the lack of a clinical skills assessment component in the National Board assessments had contributed to what I perceived as a change in focus, presumably unintended, in US medical teaching and student learning away from the patient towards the theoretical basis of medicine. Unfortunately, the validity of this assertion has never been substantiated or refuted because, as far as I know, no formal studies were ever undertaken to explore the possibility of this relationship. Of course, the NBME has now made efforts to restore the balance with the introduction to the US Medical Licensing Examination of a simulated patient-based clinical skills component. The majority of teaching was conducted in tutorial rooms and focused on theory This experience in the USA contrasted markedly with my experiences in Adelaide prior to the curriculum change described at the beginning of this commentary. Australia and the UK did not (and still do not) have national certification examinations and rely instead on accreditation procedures imposed by the General Medical Council. In the early 1980s, medical schools were required by regulation to conduct both written and traditional clinical viva voce examinations that were moderated by independent external examiners. Our students, unlike their American counterparts, were therefore not faced with national written examinations and so were constantly demanding more bedside teaching because they perceived this as the most effective strategy with which to achieve final-year success. The high-stakes nature of the clinical viva appeared to have the obverse but intended effect on student priorities despite its lack of reliability, which was unrecognised at the time. On my return to Australia, I found the winds of educational change were already sweeping across the country. A new PBL-based medical school was being established in Newcastle and my own university was introducing a more integrated curriculum. In light of the NBME research, the traditional clinical viva was at last dropped from final examinations. Unfortunately, this had the unexpected consequences outlined earlier, which paralleled what I had observed in the USA. It may be that we were advantaged in rectifying the situation by the flexibility with which we were able to change our assessment approach relatively quickly without the need to consult other stakeholders. It was also fortuitous that I had returned with some inside knowledge of the newly invented OSCE, having visited Dundee during my sabbatical, which meant we had to hand a ready-made alternative to the traditional examination format. Thus we became one of the first medical schools to introduce an OSCE into its final examinations and I believe the first to do so as an integrated examination conducted jointly by the Departments of Medicine and Surgery (and subsequently also incorporating the other clinical disciplines involved in final-year teaching). Incidentally, it also led to some ground-breaking research with new colleagues acquired in America on the psychometric characteristics of the OSCE.8 In our paper, Jaeger and I reported on the impact of this strategy on our students’ satisfaction and study habits in relation to each of the three assessment components in the final year by surveying the students shortly after graduation at the start of their intern year.1 In brief, the result showed a very positive impact of the introduction of the OSCE component (which they held in high regard and considered as a much fairer examination than the traditional viva) and an apparent rebalancing of students’ perceptions of the relative importance of the theoretical and clinical components. There was also a gratifying increase in mean class performance on clinical assessments over the next 3 years. How much this reflected actual improvement in students’ clinical skills and how much it indicated improvement in their OSCE-taking skills is hard to know. This is all part of the rich tapestry of the impact of assessment methods on student performance, which we rarely take the trouble to try and unravel. To our surprise, this work1 turned out to have an impact beyond our medical school. It appeared that no previous studies had investigated the effects of deliberately changing a curriculum assessment strategy to alter student behaviour and learning. It reinforced the view that assessments exert a dominant role in students’ lives, and not only in the ways apparent in this work.1 That said, my years of experience as a teacher, student counsellor, assessment workshop organiser and curriculum consultant lead me to conclude that the importance of this message has still not been fully understood and appreciated by many medical schools and regulatory bodies. We frequently see glaring examples of mismatches between stated objectives and both the contents of assessments and the test methods used. It appeared that no previous studies had investigated the effects of deliberately changing a curriculum assessment strategy to alter student behaviour and learning My own appreciation of the impact of assessments came from discussions about how students prepared for their examinations, both with individual students who had failed examinations and with cohorts of students after they had passed. I strongly recommend this as a litmus test-like method for finding out if students are following the faculty's curriculum outcome objectives or if, unbeknown to their teachers, they are following a different or significantly skewed ‘hidden curriculum’. Much is still to be learned about student learning and its relationship to assessment, but we know enough to be sure the latter is an important and complex relationship, and that the area deserves a lot more attention and research. Much is still to be learned about student learning and its relationship to assessment, but we know enough to be sure the latter is an important and complex relationship

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.