Field Test Evaluation of Educational Software: A Description of One Approach
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
Educational evaluators in general have traditionally recognized the needto incorporate data from potential users in designing evaluation studies. In the field of courseware evaluation, however, there has been a great deal of emphasis placed on expert judgment as a source of data for evaluating computer-based educational materials. Although courseware reviews are extremely useful, they are not substitutes for field tests;each provides a different type of information that evaluators may use in order to determinethe quality of an instructional product.This paper reports on the evaluation of a courseware designed to assist the writing of the lower-case alphabet. The main objective of the article is to demonstrate an evaluation design which provided adequate answers to our evaluation questions, allowed us to perform multiple comparisons to support our conclusions, and was also practical enough to be used in a normal classroom situation without disturbing everyday activities. Three criteria for selecting a design are presented followed by a description of the courseware evaluation.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.003 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it