Bibliographic record
Abstract
Background: Program evaluation and performance audit are distinct professional practices. Performance audit draws heavily on its financial accounting and auditing roots while evaluators have much more multidisciplinary backgrounds. Performance audit practice and development is associated with audit institutions such as National Audit Offices and internal audit while program evaluation is more associated with government departments and agencies in the public sector. The two practices are often working in the same group or organization but not necessarily together which could be mutually beneficial. Purpose: The paper examines the early contribution that Ray Rist made to resolving the methodological challenge of performance auditors and program evaluators working together, early in his career at the GAO. Setting: Independent National Audit Offices and government departments and agencies in the U.S.A. and Canada. Intervention: n/a. Research Design: The analysis of two early approaches to program evaluation to assess how they facilitated performance auditors and evaluators to work together. Data Collection and Analysis: Data were collected through an analysis of documents, publications and discussions with colleagues and associates of Ray Rist. Findings: Ray Rist worked in the group at the GAO that did evaluations and training of the multi-disciplinary staff in evaluation methodology and procedures. The approach, led by Ray, was a methodology that used multidisciplinary teams to answer specific questions. It encompassed a broader set of questions and that can more easily be used by performance auditor and program evaluators. It generally was less restricted in its application and use than other definitions of program evaluation that were based on an evaluation of a unit of analysis of a program or sub program. His early methodology transfer paper, written while Ray was at the GAO was in use for some time after he left, with numerous updates continuing with the same themes developed in the first iteration. These themes were foundational and carried into Ray’s future work, still having a strong resonance today.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.017 | 0.041 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.000 |
| Bibliometrics | 0.003 | 0.002 |
| Science and technology studies | 0.008 | 0.004 |
| Scholarly communication | 0.005 | 0.003 |
| Open science | 0.001 | 0.003 |
| Research integrity | 0.002 | 0.007 |
| Insufficient payload (model declined to judge) | 0.035 | 0.006 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".