We Can’t Hear You – You’re on Mute: Findings From a Review of Evaluation Capacity Building (ECB) Practice Online
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
Background: In her presidential address to the American Evaluation Association (AEA) in 2007, Hallie Preskill (2008) highlighted the potential role of technology to promote learning from evaluation, noting the increased use of computers, the internet, and social media as untapped ways to facilitate evaluation. More than ten years later in the context of the COVID-19 pandemic, evaluators and evaluation capacity building (ECB) practitioners found themselves needing to shift to online modalities to conduct evaluation and build capacity. The COVID-19 pandemic, technological advancements, and the rapid shift to remote work have changed our way of working (Gratton, 2021; Kane et al., 2021). Building evaluation capacity is no exception to this trend. Purpose: This study aimed to examine ways that practitioners have built evaluation capacity online or have used technology to do so, to capture lessons learned that can be applied in a COVID and post-normal context. Setting: Findings from this study can be applied in online contexts for developing evaluation capacity. Intervention: Not applicable. Research Design: The study design consisted of a rapid review of the ECB literature published from 2000 to 2019 in eight academic journals focused on evaluation research and practice. Data Collection and Analysis: Twenty-nine case applications of ECB practice that: 1) mentioned use of technology as a strategy for building evaluation capacity or 2) noted that at least one component of the ECB intervention was carried out online or virtually were reviewed for this study. Quantitative data were analyzed via descriptive statistics. Qualitative data were coded in MAXQDA using conventional content analysis (Hsieh & Shannon, 2005). Findings: More diverse online interventions have increased over time. Less than half (45%) of ECB interventions made use of both asynchronous and synchronous strategies for building capacity while more than one-third (38%) made use of asynchronous only strategies. Key barriers to implementing ECB strategies online included lack of social connections to other participants during the capacity building activity, technical malfunctions, lack of access to or familiarity with the technology in use, and limited resources for carrying out evaluation activities. Key facilitators for enhancing implementation included facilitating participant interaction and relationship-building both on and off-line, tailoring ECB activities to participant work contexts, and providing tutorials for accessing and using the technology in play.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.092 | 0.044 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.003 | 0.001 |
| Bibliometrics | 0.002 | 0.003 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.001 |
| Open science | 0.001 | 0.000 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.002 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it