Evaluation Methods, Indicators, and Outcomes in Learning Health Systems: Protocol for a Jurisdictional Scan
Notice bibliographique
Résumé
BACKGROUND: In learning health systems (LHSs), real-time evidence, informatics, patient-provider partnerships and experiences, and organizational culture are combined to conduct "learning cycles" that support improvements in care. Although the concept of LHSs is fairly well established in the literature, evaluation methods, mechanisms, and indicators are less consistently described. Furthermore, LHSs often use "usual care" or "status quo" as a benchmark for comparing new approaches to care, but disentangling usual care from multifarious care modalities found across settings is challenging. There is a need to identify which evaluation methods are used within LHSs, describe how LHS growth and maturity are conceptualized, and determine what tools and measures are being used to evaluate LHSs at the system level. OBJECTIVE: This study aimed to (1) identify international examples of LHSs and describe their evaluation approaches, frameworks, indicators, and outcomes; and (2) describe common characteristics, emphases, assumptions, or challenges in establishing counterfactuals in LHSs. METHODS: A jurisdictional scan, which is a method used to explore, understand, and assess how problems have been framed by others in a given field, will be conducted according to modified PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. LHSs will be identified through a search of peer-reviewed and gray literature using Ovid MEDLINE, EBSCO CINAHL, Ovid Embase, Clarivate Web of Science, PubMed non-MEDLINE databases, and the web. We will describe evaluation approaches used both at the LHS learning cycle and system levels. To gain a comprehensive understanding of each LHS, including details specific to evaluation, self-identified LHSs will be included if they are described according to at least 4 of 11 prespecified criteria (core functionalities, analytics, use of evidence, co-design or implementation, evaluation, change management or governance structures, data sharing, knowledge sharing, training or capacity building, equity, and sustainability). Search results will be screened, extracted, and analyzed to inform a descriptive review pertaining to our main objectives. Evaluation methods and approaches, both within learning cycles and at the system level, as well as frameworks, indicators, and target outcomes, will be identified and summarized descriptively. Across evaluations, common challenges, assumptions, contextual factors, and mechanisms will be described. RESULTS: As of October 2024, the database searches described above yielded 3503 citations after duplicate removal. Full-text screening of 117 articles is complete, and 49 articles are under analysis. Results are expected in early 2025. CONCLUSIONS: This research will characterize the current landscape of LHS evaluation approaches and provide a foundation for developing consistent and scalable metrics of LHS growth, maturity, and success. This work will also serve to identify opportunities for improving the alignment of current evaluation approaches and metrics with population health needs, community priorities, equity, and health system strategic aims. TRIAL REGISTRATION: Open Science Framework b5u7e; https://osf.io/b5u7e. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/57929.
Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.
Comment cette classification a été obtenuedéplier
Prédiction machine sur la base complète
Imitation des enseignantsNi prévalence calibrée, ni vérité terrain. Validation humaine à venir. Le volet Gemma est une étiquette directe du modèle pour chaque travail de la base, lue sur la notice réduite au titre. Le volet Codex est un classifieur appris des 10 348 étiquettes directes de Codex et calibré sur les taux pondérés de l'échantillon; les champs sans appui suffisant ne portent aucun appel Codex. Le mode candidate est l'union des deux volets; le consensus est leur intersection. Ces sorties portent le statut machine_predicted_unvalidated et ne sont pas des étiquettes humaines.
Scores du classifieur distillé par catégorie (deux têtes)
| Catégorie | Codex | Gemma |
|---|---|---|
| Métarecherche | 0,204 | 0,262 |
| Méta-épidémiologie (sens strict) | 0,005 | 0,005 |
| Méta-épidémiologie (sens large) | 0,011 | 0,012 |
| Bibliométrie | 0,013 | 0,020 |
| Études des sciences et des technologies | 0,007 | 0,006 |
| Communication savante | 0,009 | 0,009 |
| Science ouverte | 0,006 | 0,007 |
| Intégrité de la recherche | 0,011 | 0,010 |
| Charge utile insuffisante (le modèle a refusé de juger) | 0,105 | 0,018 |
Scores machine (provisoires)
Les deux têtes enseignantes du modèle étudiant, lues sur ce travail. Un score ordonne la base pour la relecture; il n'affirme jamais une catégorie, et le statut de validation accompagne chaque rangée tel quel.
Scores de référence d'un modèle non mature (critères de maturité non atteints, 7 itérations). Un score ordonne; il n'affirme jamais une catégorie.
score_only:v0-immature-baseline · tel quel depuis la passe de notation : score_only signifie que le nombre peut ordonner les travaux, et qu'aucune étiquette de catégorie n'en découleClassification
machine, non validéePrédiction automatique; l’étiquette directe de Gemma et le classifieur distillé Codex s’accordent sur ce qui est montré ici.
Le détail, modèle par modèle et score par score, se trouve en fin de page sous « Comment cette classification a été obtenue ».