Embedding the Emerging Evaluator, Evaluation as Learning in Community Arts
Bibliographic record
Abstract
Sistema Kingston (SK) is an El-Sistema inspired intensive aQer-school music program that focuses on positive social change through the pursuit of free, group-centred music instruction. I have been invested with the program personally and professionally since its inception in 2015. Following suit of similar programs across Canada and the world, SK initiated an evaluation process to address concerns surrounding the program's future consistency and sustainability. Though some may not consider consistency to be preferable if the status quo is not always effectively serving students, families, and communities, I interpret this concern raised by the program director to refer predominantly to a current lack of transferable continuity and flow between the teaching methods and curriculum content actioned by transient teaching team members. Combining my scholarly interest in evaluation and assessment, with a desire to support SK, this project saw me as an embedded external formative evaluator who developed tools and instruments to build capacity within SK and collect data to answer questions prioritized by SK. The process evaluation conducted using developmental evaluation techniques unearthed rich qualitative data, that when analyzed by thematic coding revealed four overarching areas of program consideration around which recommendations could be generated, including Representation and Voice, Staff Recruitment, Curriculum, Assessment and Group Structure, and an effusive SK Genie. The evaluation questions were addressed by elaborating on the recommendations, and an evaluation report offered to the directorship of Sistema Kingston, in the hope that the knowledge will be mobilized to secure the future of the program. By reflecting on the experience of designing and undertaking an evaluation process, extracting, and analyzing data, and reporting findings to a community partner, I have learnt and grown as a researcher and academic, as well as progressing my emergence as an evaluator.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.308 | 0.247 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.005 | 0.003 |
| Science and technology studies | 0.008 | 0.045 |
| Scholarly communication | 0.030 | 0.017 |
| Open science | 0.003 | 0.020 |
| Research integrity | 0.005 | 0.011 |
| Insufficient payload (model declined to judge) | 0.004 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".