Designing a program evaluation for a medical‐dental service for adults with autism and intellectual disabilities using the RE‐AIM framework
Bibliographic record
Abstract
INTRODUCTION: Robust evaluation of service models can improve the quality and efficiency of care while articulating the models for potential replication. Even though it is an essential part of learning health systems, evaluations that benchmark and sustain models serving adults with developmental disabilities are lacking, impeding pilot programs from becoming official care pathways. Here, we describe the development of a program evaluation for a specialized medical-dental community clinic serving adults with autism and intellectual disabilities in Montreal, Canada. METHOD: Using a Participatory Action-oriented approach, researchers and staff co-designed an evaluation for a primary care service for this population. We performed an evaluability assessment to identify the processes and outcomes that were feasible to capture and elicited perspectives at both clinical and health system levels. The RE-AIM framework was used to categorize and select tools to capture data elements that would inform practice at the clinic. RESULTS: We detail the process of conceptualizing the evaluation framework and operationalizing the domains using a mixed-methods approach. Our experience demonstrated (1) the utility of a comprehensive framework that captures contextual factors in addition to clinical outcomes, (2) the need for validated measures that are not cumbersome for everyday practice, (3) the importance of understanding the functional needs of the organization and building a sustainable data infrastructure that addresses those needs, and (4) the need to commit to an evolving, "living" evaluation in a dynamic health system. CONCLUSIONS: Evaluation employing rigorous patient-centered and systems-relevant metrics can help organizations effectively implement and continuously improve service models. Using an established framework and a collaborative approach provides an important blueprint for a program evaluation in a learning health system. This work provides insight into the process of integrating care for vulnerable populations with chronic conditions in health care systems and integrated knowledge generation processes between research and health systems.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.232 | 0.154 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.003 |
| Bibliometrics | 0.005 | 0.003 |
| Science and technology studies | 0.005 | 0.005 |
| Scholarly communication | 0.007 | 0.005 |
| Open science | 0.004 | 0.008 |
| Research integrity | 0.002 | 0.002 |
| Insufficient payload (model declined to judge) | 0.004 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; the direct Gemma label and the distilled Codex classifier agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".