Evaluation Strategies for Understanding Experiences With Virtual Care in Canada: Mixed Methods Study
Bibliographic record
Abstract
BACKGROUND: Virtual care was rapidly integrated into pediatric health services during the COVID-19 pandemic. While virtual care offers many benefits, it is necessary to better understand the experiences of those who receive, deliver, and coordinate virtual care in order to support sustainable, high-quality, and patient-centered health care. To date, methods implemented to evaluate users' experiences of virtual care have been highly variable, making comparison and data synthesis difficult. OBJECTIVE: This study aims to describe evaluation strategies currently used to understand personal experiences with pediatric virtual care in Canada. METHODS: In this mixed methods environmental scan, we first distributed a web-based questionnaire to clinical, research, and operational leaders delivering and evaluating pediatric virtual care in Canada. The questionnaire collected information about how experiences with virtual care have been or are currently being evaluated and whether these evaluations included the perspectives of children or youth, families, providers, or support staff. Second, respondents were asked to share the questions they used in their evaluations, and a content analysis was performed to identify common question categories. Third, we conducted semistructured interviews to further explore our respondents' evaluation experiences across 4 domains-evaluation approaches, distribution methods, response rates, and lessons learned-and interest in a core set of questions for future evaluations. RESULTS: There were 72 respondents to the web-based questionnaire; among those who had conducted an evaluation, we identified 15 unique evaluations, and 14 of those provided a copy of the tools used to evaluate virtual care. These evaluations measured the virtual care experiences of parents or caregivers (n=15, 100%), children or youth (n=11, 73%), health care providers (n=11, 73%), and support staff (n=4, 27%). The most common data collection method used was electronic questionnaires distributed by email. Two respondents used validated tools; the remainder modified existing tools or developed new tools. Content analysis of the 14 submitted questionnaires revealed that the most common questions were about overall participant satisfaction, the comparison of virtual care to in-person care, and whether participants would choose virtual care options in the future. Interview findings indicate respondents frequently relied on methods used by peers and that a standardized, core set of questions to evaluate experiences with virtual care would be helpful to improve evaluation practices and support pediatric health care delivery. CONCLUSIONS: At our institution and elsewhere in Canada, experiences with pediatric virtual care have been evaluated using a variety of methods. A more consistent evaluation approach using standardized tools may enable more regular comparisons of experiences with virtual care and the synthesis of findings across health care settings. In turn, this may better inform our approach to virtual care, improve its integration into health systems, and facilitate sustainable, high-quality, patient-centered care.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.129 | 0.126 |
| Meta-epidemiology (narrow) | 0.002 | 0.002 |
| Meta-epidemiology (broad) | 0.002 | 0.002 |
| Bibliometrics | 0.010 | 0.015 |
| Science and technology studies | 0.018 | 0.006 |
| Scholarly communication | 0.011 | 0.004 |
| Open science | 0.004 | 0.009 |
| Research integrity | 0.002 | 0.002 |
| Insufficient payload (model declined to judge) | 0.004 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".