A confirmatory approach to differential item functioning on an ESL reading assessment
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
In this article, I describe a practical application of the Roussos and Stout (1996) multidimensional analysis framework for interpreting group performance differences on an ESL reading proficiency test. Although a variety of statistical methods have been developed for flagging test items that function differentially for equal ability examinees from different ethnic, linguistic, or gender groups, the standard differential item functioning (DIF) detection and review procedures have not been very useful in explaining why DIF occurs in the flagged items (Standards for Educational and Psychological Testing 1999). To address this problem, Douglas, Roussos and Stout (1996) developed a confirmatory approach to DIF, which is used to test DIF hypotheses that are generated from theory and substantive item analyses. In the study described in this paper, DIF and differential bundle functioning (DBF) analyses were conducted to determine whether groups of reading test items, classified according to a bottom-up, top-down reading strategy framework, functioned differentially for equal ability Arabic and Mandarin ESL learners. SIBTEST (Stout and Roussos, 1999) analyses revealed significant systematic group differences in two of the bottom-up and two of the top-down reading strategy categories. These results demonstrate the utility of employing a theoretical framework for interpreting group differences on a reading test.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.000 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.001 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it