Investigating construct representativeness and linguistic equity of automated oral reading fluency assessment with prosody
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
While the automated assessment of oral reading fluency (ORF) using accuracy and speech rate has proliferated, expressiveness of speech, as measured by prosodic features, has been neglected due to its inherent complexity and lack of technological resources. Despite the potential benefits of burgeoning technology for assessing hard-to-measure constructs such as ORF, insensitivity to linguistic diversities threatens valid score interpretations and fair use for all learners. The present study investigated the potential benefits of developing an automated, prosody-inclusive ORF assessment in a post-secondary education setting involving many English language learners (ELLs). The analysis focused on three ways the inclusion of prosody may improve automated ORF assessments: by reducing bias against ELLs, improved prediction of reading comprehension, and improved diagnostic information. Data were analyzed by comparing two different scoring outcomes, the traditional ORF measure and a new prosody-inclusive outcome score, comparing these measures across language background. Results showed that the inclusion of prosody improves automated ORF assessment by reducing discrepancies between ELLs and English first language students caused by automated speech recognition inaccuracies, leads to better prediction of reading comprehension with ELLs, and provides meaningful diagnostic information. Detailed descriptions of the models, their relevance, and implications for the language testing community are discussed.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.001 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.001 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it