Flying blind, or going with the flow?: Using constructivist evaluation to manage the unexpected in the GraniteNet project
Bibliographic record
Abstract
The GraniteNet Project is a research and development collaboration between the University of Southern Queensland, Australia, and the community of Stanthorpe – a rural community of just over 10,000 people located within the university’s regional catchment area. The vision of this Community Informatics project, which commenced in 2007 and is now in its third phase, is the development of a sustainable community designed, owned and managed web portal that will support Stanthorpe’s development as a ‘learning community’. With funding from the State Government, the GraniteNet Board commissioned an evaluation of the second phase of the project which focussed on the design, development and trial of an incubator community portal environment, a portal governance framework and community engagement strategy. Participatory Action Research (PAR) and constructivist (or “Fourth Generation”) evaluation methodologies were adopted to guide the evaluation with the aims of documenting the project, establishing an evidence base to inform future decision-making, identifying and exploring significant contextual factors impacting on the project, evaluating the effectiveness of the models and processes used to guide the project, and building a culture of evaluation that would help to ensure ongoing review and critical reflection on progress. The evaluation design encompassed formative, summative and research evaluation. This paper reports the evaluation processes and outcomes, with a focus on exploring the ways in which these methodologies can be used to help Community Informatics researchers and practitioners learn from and about the unexpected and unanticipated in a field where learning through experimentation is the name of the game, imagination, creativity and collaborative design the keys to innovation and transformation, and where more traditional evaluation methodologies are becoming increasingly irrelevant.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.469 | 0.375 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.002 | 0.001 |
| Bibliometrics | 0.009 | 0.007 |
| Science and technology studies | 0.013 | 0.033 |
| Scholarly communication | 0.031 | 0.020 |
| Open science | 0.007 | 0.021 |
| Research integrity | 0.004 | 0.009 |
| Insufficient payload (model declined to judge) | 0.002 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".