Evaluating for learning and sustainability (ELS) framework: a realist synthesis
Bibliographic record
Abstract
Abstract Background Learning Health Systems (LHS), in which continuous and equitable improvements support optimization of healthcare practices, outcomes, experience, and costs, offer enormous potential for health system transformation. Within the LHS model, evaluation of health innovations assists in question identification, data collection, and targeted action, which facilitates continuous improvement. Evaluation that catalyzes learning may contribute to health innovation implementation, refinement, and sustainability, however, there is little consensus as to why certain evaluations support learning, while others impede it. Methods Embedded in the implementation science literature, we conducted a realist synthesis to understand evaluative contextual factors and underlying mechanisms that best support health system learning and sustainable implementation of innovations. We sought to understand whether evaluations can ‘work’ to support learning and sustainability, in which contexts, for whom, and why. Working with an Expert Committee comprised of leaders in evaluation, innovation, sustainability, and realist methodology, we followed a five-stage process of: 1. Scoping the Review, 2. Building Theories, 3. Identifying the Evidence, 4. Evidence Selection and Appraisal, and 5. Data Extraction and Synthesis. Our Review Team and Expert Committee participated in iterative cycles of results interpretation and feedback. Results Our synthesis includes 60 articles capturing the mechanisms and contextual factors driving learning and sustainability through evaluation. We found that evaluations that support learning and sustainability incorporate favourable organizational preconditions and focus on implementing rapid cyclical feedback loops that contribute to a culture of innovation and evaluation sustainability. Our findings have been organized into 6 Context-Mechanism-Outcome Configurations (CMOCs): 1. Embracing Risk & Failure; 2. Increasing Capacity for Evaluation; 3. Co-Producing Evaluation; 4. Implementing Learning Feedback Loops; 5. Creating Sustainability Culture; and 6. Becoming a Learning Organization. We have also translated findings into a series of Action Strategies for evaluation implementation to support health systems learning and sustainability. Conclusions We identified key contextual factors and underlying mechanisms that make evaluations ‘work’ (or ‘not work’) to support learning and sustainability. Findings support the operationalization of LHS by translating CMOCs into Action Strategies for those tasked with completing evaluations with a view toward health system learning and innovation sustainability.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.058 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.001 | 0.001 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.001 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.186 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".