The Design and Use of Assessment Frameworks in Digital Curation
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
To understand and improve their current abilities and maturity, organizations use diagnostic instruments such as maturity models and other assessment frameworks. Increasing numbers of these are being developed in digital curation. Their central role in strategic decision making raises the need to evaluate their fitness for this purpose and develop guidelines for their design and evaluation. A comprehensive review of assessment frameworks, however, found little evidence that existing assessment frameworks have been evaluated systematically, and no methods for their evaluation. This article proposes a new methodology for evaluating the design and use of assessment frameworks. It builds on prior research on maturity models and combines analytic and empirical evaluation methods to explain how the design of assessment frameworks influences their application in practice, and how the design process can effectively take this into account. We present the evaluation methodology and its application to two frameworks. The evaluation results lead to guidelines for the design process of assessment frameworks in digital curation. The methodology provides insights to the designers of the evaluated frameworks that they can consider in future revisions; methodical guidance for researchers in the field; and practical insights and words of caution to organizations keen on diagnosing their abilities.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.005 | 0.006 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.001 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.002 | 0.070 |
| Open science | 0.001 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it