Validation of a framework for evaluating knowledge mobilization strategies: A Delphi method approach.
Bibliographic record
Abstract
Background: A growing number of knowledge-oriented organizations, such as granting agencies, governments, public organizations, universities, and health authorities, are investing considerable resources to increase the use of research knowledge to improve professional practice, decision making, and public policy. The proliferation of research on knowledge mobilization (KMb) over the past two decades has deepened our understanding of the dynamics of this process and of the factors that can impede its deployment, such as knowledge users’ capabilities (their beliefs, capacity to absorb knowledge, etc.), contextual conditions (resources, leadership, facilitating factors, etc.), and the availability of effective mobilization strategies (frequency, implementation, fit with context). However, yet, few good-quality studies have evaluated the impacts of KMb, such that we still know too little about the effectiveness of the different strategies and the contextual conditions in which they may be effective. This is problematic, in that their development cannot be fully grounded in empirical evidence. In fact, their evaluation is complicated by the virtual absence of evaluation tools and validated indicators that would allow organizations to assess the impacts of their KMb strategies. Moreover, the difficulty that these organizations experience in relation to evaluation (due to lack of expertise and resources) is a concern that has been raised many times. Aims: This study will address this expressed need to improve organizations’ capacity to conduct KMb evaluation studies. Using a collaborative co-construction approach with key actors in KMb, our aim is to design and validate an integrative and operational framework for the evaluation of KMb strategies in the social domain. Design/approach: a first step in this project, we conducted a scoping review of frameworks and theories commonly used to evaluate KMb strategies. 71 articles were selected from this scoping review. Our analyses of these articles, we identified four potentially relevant dimensions for planning and evaluation: the context, implementation process, effects, and impacts of these strategies. Using the Delphi approach, a consultation has been undertaken to enrich and validate the dimensions of this framework developed after a scoping review. Results: This paper presents the results of the Delphi consultation with an international panel of experts working in the field of knowledge mobilization. This evaluation exercise should lead to a validation of the framework components and potential indicators to be considered when evaluating knowledge mobilization strategies.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.221 | 0.178 |
| Meta-epidemiology (narrow) | 0.003 | 0.002 |
| Meta-epidemiology (broad) | 0.002 | 0.003 |
| Bibliometrics | 0.015 | 0.009 |
| Science and technology studies | 0.006 | 0.007 |
| Scholarly communication | 0.006 | 0.005 |
| Open science | 0.005 | 0.013 |
| Research integrity | 0.003 | 0.003 |
| Insufficient payload (model declined to judge) | 0.005 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; the direct Gemma label and the distilled Codex classifier agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".