Production and use of rapid responses during the COVID-19 pandemic in Quebec (Canada): perspectives from evidence synthesis producers and decision makers
Bibliographic record
Abstract
BACKGROUND: The COVID-19 pandemic has required evidence to be made available more rapidly than usual, in order to meet the needs of decision makers in a timely manner. These exceptional circumstances have caused significant challenges for organizations and teams responsible for evidence synthesis. They had to adapt to provide rapid responses to support decision-making. This study aimed to document (1) the challenges and adaptations made to produce rapid responses during the pandemic, (2) their perceived usefulness, reported use and factors influencing their use and (3) the methodological adaptations made to produce rapid responses. METHODS: A qualitative study was conducted in 2021 with eight organizations in the health and social services system in Quebec (Canada), including three institutes with a provincial mandate. Data collection included focus groups (n = 9 groups in 8 organizations with 64 participants), interviews with decision makers (n = 12), and a document analysis of COVID-19 rapid responses (n = 128). A thematic analysis of qualitative data (objectives 1 and 2) and a descriptive analysis of documents (objective 3) were conducted. RESULTS: The results highlight the teams and organizations' agility to deal with the many challenges encountered during the pandemic (e.g., increased their workloads, adoption of new technological tools or work processes, improved collaboration, development of scientific monitoring, adaptation of evidence synthesis methodologies and products). The challenge of balancing rigor and speed was reported by teams and organizations. When available at the right time, rapid responses have been reported as a useful tool for informing or justifying decisions in a context of uncertainty. Several factors that may influence their use were identified (e.g., clearly identify needs, interactions with producers, perceived rigor and credibility, precise and feasible recommendations). Certain trends in the methodological approaches used to speed up the evidence synthesis process were identified. CONCLUSIONS: This study documented rapid responses producers' experiences during the COVID-19 pandemic in Quebec, and decision makers who requested, consulted, or used these products. Potential areas of improvements are identified such as reinforce coordination, improve communication loops, clarify guidelines or methodological benchmarks, and enhance utility of rapid response products for decision makers.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Direct model labels (unvalidated)
Per-model category and study-design labels from the labeling rounds. They are machine output, unvalidated, and the disagreement between models ships as data. No study design here is MEDLINE-validated yet.
| Model arm | Categories | Study design | Confidence |
|---|---|---|---|
| gemma | Metaresearch Domain: Methods · Genre: Empirical About the Canadian research system: yes · About a Canadian topic: yes | Qualitative | low |
| gpt | MetaresearchScholarly communication Domain: Methods · Genre: Empirical About the Canadian research system: yes · About a Canadian topic: yes | Observational | medium |
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.022 | 0.105 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.001 | 0.001 |
| Science and technology studies | 0.002 | 0.001 |
| Scholarly communication | 0.000 | 0.001 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedLabeled directly by 2 models reading the full record.
The models disagree on parts of this classification; every voice is preserved in the section at the end of the page.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".