Evaluations of Educational Interventions: Getting Them Published and Increasing Their Impact

Advancing the evidence base that informs educational research and practice must include studies with both randomized and nonrandomized designs.1–3 Although experimental designs produce results with a high level of internal validity, randomization and strict controls are not always feasible when evaluating the impact of an educational intervention, and experimental study designs are not suited for answering all evaluation questions, especially when the aim is to produce practical knowledge for immediate use.4 Many reports of evaluations of educational interventions—especially those employing nonrandomized designs, which have methods and reporting conventions that are less standardized—are submitted to JCEHP with significant shortcomings and, as a result, go unpublished or require extensive revisions. In some cases, this reflects shortcomings in the study design; in others, the problem lies more with how the study was initially written up. This is unfortunate for several reasons, not the least of which is the investment in time and effort authors invest in preparing manuscripts for submission. There is currently little guidance specifically for reporting evaluations of continuing education interventions. There are guidelines on reporting innovations in medical school curricula,5 quality improvement projects in health care organizations,6 nonrandomized evaluations of public health interventions,1 and evaluation studies of health promotion and disease prevention programs.7 There are also general guidelines for reporting randomized controlled trials8 and complex interventions for improving health.3 However, none of these provides a comprehensive guide for continuing education for health professionals (CEHP) evaluations. Providing a comprehensive set of guidelines for reporting evaluations of CEHP interventions is beyond the scope of this editorial; however, we would like to highlight some of the common errors authors make and suggest ways to increase the likelihood that a manuscript will be accepted for publication, make an evaluation report more useful to its readers, and allow the reader to make a more informed assessment of the rigor of the study and the credibility of its findings. It is also our hope that through a process of backward planning, these suggestions will contribute to improved evaluation design and implementation as well. First, some background. Educational interventions are typically complex. Complexity may be due to the number of interacting components within an intervention, the difficulty of the behaviors required of those delivering or receiving the intervention, the number and variability of the intended outcomes, and the degree of tailoring of the intervention permitted.3 To the extent that the intervention has as its outcome improving clinical practice or practice redesign, it becomes a complex intervention in a complex system, resulting in multiple and sometimes unpredictable interactions between the components of the intervention and the environment in which it is implemented.9 There are several implications of this increase in complexity for CEHP evaluations: A good theoretical understanding is needed of how the intervention causes change, so that weak links in the causal chain can be identified and strengthened. Lack of effect may reflect implementation failure (or teething problems) rather than genuine ineffectiveness; a thorough process evaluation is needed to identify implementation problems. A single primary outcome may not provide sufficient data; a range of measures will be needed and unintended consequences picked up where possible. Ensuring strict standardization may be inappropriate; the intervention may work better if a specified degree of adaptation to local settings is allowed for in the protocol.3 These realities pose significant challenges in the design and reporting of evaluations. Among others, they mean that we need to broaden the scope of our evaluation inquiries, asking not only does an intervention work but also how it works and in what settings. Understanding the relationships among the various components of the intervention, the observed outcomes, and the context is essential.1,3,10 With this as a backdrop, here are some problems frequently encountered in reports of nonrandomized evaluations submitted to JCEHP, organized around the IMRaD structure (introduction, methods, results, and discussion) used for many evaluation reports. Many of the suggestions we offer for addressing these problems are relevant to reports of experimental evaluation studies as well. INTRODUCTION A focus on gaps in clinical practice and patient outcomes at the local level only. Many evaluations focus on interventions aimed at solving problems in one health care organization or small geographic region. However, although most of JCEHP’s readers are in the United States and Canada, the journal has an international readership, making it important to demonstrate that there is a need for the intervention at the national or even international level as well.5 Lack of a rationale establishing the need for an educational intervention to close the gaps identified. In addition to describing the problem, how it was discovered, and the intended improvement,6 the introduction should also indicate how it was determined that the improvement required an educational intervention. One of the criticisms of CEHP as a field of practice is that we often assume that gaps between desired and actual clinical practice are due to gaps in knowledge and skills, when instead they may have other causes, such as lack of resources or incompatible systems.11 This makes it especially important to offer a rationale for the use of an educational intervention that identifies a clear learning need12—a gap in knowledge, skills, or attitudes. Lack of a successful argument for the importance of the contribution to the literature. Although nonrandomized evaluations are more often used in effectiveness studies where the primary goal is to produce practical knowledge rather than contribute to the scientific literature,4 it is good educational and scholarly practice to not “reinvent the wheel” and build on the efforts of others to develop similar educational interventions as reported in both the published and, since many evaluation reports are not published via traditional channels, grey literature.13 Furthermore, journal pages are valuable real estate, which means that reviewers and editors need convincing that the evaluation makes a significant contribution to the field’s knowledge base. Both concerns are addressed by a concise summary of the literature on prior attempts to develop interventions addressing the clinical problem.5 This review should answer the question of what is already known and not known from both a scientific and practical perspective and will make it possible to more explicitly state the unique contribution of the study and identify the evidence base for the intervention.3 METHODS Intervention Inadequate description of the intervention. A frequent complaint of investigators doing systematic reviews on the effectiveness of educational interventions is that the intervention is not well described, making it difficult to draw conclusions about the relationship between educational approaches (eg, lecture, case discussion, academic detailing) and outcomes. A sufficiently detailed description of the intervention also facilitates its adoption or adaptation in other settings. Reznich and Anderson5 suggest that the description should include the intended outcomes, content, educational methods, sequencing, and time allocation, with has an emphasis on components that are innovative. Although word limits constrain the length of intervention descriptions, authors have the option of submitting supplemental material that is referenced in the print version of the article but published only with the online version. JCEHP has no restrictions on the length of these materials. Inadequate description of the intervention development process. Authors will often describe the intervention and omit information about how the intervention was developed. The rationale behind the intervention is important background and should provide insight into how the evidence base informed the design, identify key programmatic decisions made,5 and relate how contextual factors influenced the design and other dimensions of the planning process. Little or no description of the context. As Pawson and Tilley observed: Programs are always introduced into pre-existing social contexts and...these prevailing social conditions are of crucial importance when it comes to explaining the successes and failures of social programs.10 Relevant aspects of the setting or settings in which intervention and change are expected to take place should be described,5,6 as should their predicted influence on the relationship between the intervention and desired outcomes (eg, will they enable or disable the mechanisms of change?).10 Lack of a theoretical framework describing how the program is expected to function. Intervention descriptions should not only include the components of the intervention and the intended outcomes, but also the mechanisms that link them. What are the causal pathways14 or program theories4 that describe how the intervention is expected to result in the expected outcomes? To the extent that the evaluation approach examines evidence assessing whether those mechanisms operated as anticipated,6 it can lend internal validity to the study (strengthening the argument that any outcomes observed were or were not attributable to the intervention). Evaluation Poorly defined or absent evaluation questions. Many evaluation reports provide only broad statements of purpose, such as “evaluate the impact of the intervention.” Evaluation questions provide the foundation of the inquiry and are an essential component for explaining how and why an intervention is being examined.15 Lack of methodological detail. Enough information should be included in the evaluation methodology section to allow the reader to assess the rigor of the evaluation and credibility of the findings. Details should include how and when data were collected and analyzed. Elements commonly omitted or inadequately described are details of the instruments used to collect data, how instruments were developed, efforts made to establish the reliability and validity of measures, when the data were collected and by whom, and who did the analysis and how any qualitative data were analyzed. Information about recruitment of participants in the study is especially important to assess external validity (generalizability). Little or no explicit identification of strategies used to strengthen internal validity (integrity of the data) and external validity (generalizability) of the study.6 A randomized controlled trial (RCT) may not be practical or may be incapable of addressing the evaluation questions; however, there are a number of alternate study designs and other strategies that can be used to strengthen the study. Actions taken to enhance rigor such as training in data collection, use of theory, assessing mechanisms linking the intervention and outcomes, and triangulation should be described. No evidence that ethical considerations were addressed. One constant across all evaluations of educational interventions is that they involve human subjects. Although many evaluations qualify for an exemption under US rules, investigators still have an obligation to protect the rights of the participants in the evaluation. Evidence should be provided that ethical issues were considered and addressed; such evidence is now a requirement for all manuscripts submitted to JCEHP.6 RESULTS Incomplete reporting of challenges and problems encountered. Most evaluations read as if the planning and implementation process were unproblematic and only positive outcomes were realized. Reports should provide summaries of both the most important successes and challenges.6 Because complex interventions inevitably produce unintended outcomes that may significantly augment the positive impacts of the intervention or detract from any gains that are realized, any important unintended outcomes should also be reported.3 An assumption that the program as implemented was the same as the program as it was intended. Variability across sites and faculty, unanticipated barriers to implementation, and the need to adapt the intervention as the program plays out are common in evaluations conducted in uncontrolled, real-world settings. Without a description of how the intervention was implemented and adaptations made after it was begun, if the intervention fails and there is no description of how the intervention was implemented or of adaptations made after it was begun, there is no way to know if it was due to a program failure or implementation failure.16 The results section should describe when and how the intervention was implemented as compared to what was initially intended (fidelity evaluation).5 DISCUSSION Overly broad conclusions. Despite the limitations of nonrandomized evaluations of interventions, it is not uncommon for authors to assert that any results were attributable to the intervention. A more satisfactory approach would be to provide a critical examination of the outcomes, including an assessment of the strength of association among observed changes, intervention, and contextual factors.6 Limitations of the study and alternative explanations for the findings should be addressed as well. Clear distinctions should be made among objective findings, judgments, and speculation; implications should be described with appropriate caveats.7 Missed opportunities to share practical lessons learned. The purpose of any study is to learn, and the occasion of an evaluation can result in learning at multiple levels, both within and around the scope defined by the study. Practical lessons learned by the investigators such as how the intervention might be revised in the next iteration5 and insights into how to make both the intervention and evaluation planning process more effective are outcomes worth reporting. This editorial draws on editorial experience, our own practice as evaluators, and the evaluation literature to provide suggestions on how to make reports of evaluations more valuable to JCEHP’s readers, allow readers to more accurately assess the credibility of the findings, and improve the likelihood of acceptance for publication. However, we believe that there remains a need for a more formal and considered set of guidelines for reporting on evaluations of educational interventions aimed at health professionals, developed, perhaps, by an interprofessional group of investigators and editors. In addition, our experience also suggests there is a need for continued efforts at capacity development around evaluation of continuing education interventions in the health care field. The quality of evaluation reports and the contributions they make to research and practice are inextricably linked to the quality of evaluation design and implementation. There are some well-developed approaches that offer design guidance, including the RE-AIM framework,17,18 realistic evaluation,10 and theory-driven outcome evaluation,4 that appear to have potential application. These warrant closer attention from the CEHP community so that we may explore if and how they might enhance our collective efforts to find innovative and effective means for improving care, improving health, and reducing costs.

Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.