Evaluation2 – Evaluating the national evaluation system in South Africa: What has been achieved in the first 5 years?
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
Background: South Africa has pioneered national evaluation systems (NESs) along with Canada, Mexico, Colombia, Chile, Uganda and Benin. South Africa’s National Evaluation Policy Framework (NEPF) was approved by Cabinet in November 2011. An evaluation of the NES started in September 2016.Objectives: The purpose of the evaluation was to assess whether the NES had had an impact on the programmes and policies evaluated, the departments involved and other key stakeholders; and to determine how the system needs to be strengthened.Method: The evaluation used a theory-based approach, including international benchmarking, five national and four provincial case studies, 112 key informant interviews, a survey with 86 responses and a cost-benefit analysis of a sample of evaluations.Results: Since 2011, 67 national evaluations have been completed or are underway within the NES, covering over $10 billion of government expenditure. Seven of South Africa’s nine provinces have provincial evaluation plans and 68 of 155 national and provincial departments have departmental evaluation plans. Hence, the system has spread widely but there are issues of quality and the time it takes to do evaluations. It was difficult to assess use but from the case studies it did appear that instrumental and process use were widespread. There appears to be a high return on evaluations of between R7 and R10 per rand invested.Conclusion: The NES evaluation recommendations on strengthening the system ranged from legislation to strengthen the mandate, greater resources for the NES, strengthening capacity development, communication and the tracking of use.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.208 | 0.009 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.001 | 0.004 |
| Science and technology studies | 0.001 | 0.000 |
| Scholarly communication | 0.003 | 0.003 |
| Open science | 0.002 | 0.000 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.004 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it