Evaluating the Explanation Inference of\na High-Stakes French Listening Test:\nAn Argument-Based Perspective
Bibliographic record
Abstract
Cet article repose sur l’approche de la validation basée sur l’argumentation pour rassembler et évaluer des preuves liées au construit (c’est-à-dire l’inférence d’explication) d’un test à enjeux critiques. Les données proviennent de la composante de compréhension orale d’un test de français utilisé pour l’immigration au Canada à travers la province du Québec. Un panel d’experts ayant des formations variées en linguistique appliquée a examiné et associé les items de deux versions opérationnelles du test à quatre sous-compétences de compréhension orale recensées dans des sources sélectionnées de la théorie de la compréhension orale en langue seconde. Sur la base des recommandations du panel d’experts, deux modèles factoriels confirmatoires ont été ajustés aux données de réponse des candidats aux tests. Les modèles se sont bien ajustés aux données, permettant de soutenir l’inférence d’explication, mais suggérant une sous-représentation du construit pour l’une des versions évaluées. L’approche de la validation basée sur l’argumentation a fourni des lignes directrices pour évaluer la représentation du construit des tests, en suggérant des recommandations éclairantes sur comment organiser les preuves du construit dans une perspective d’argumentation. Les retombées de cette recherche sont discutées en ce qui concerne l’opérationnalisation de la validation basée sur l’argumentation dans des contextes à enjeux critiques. Abstract: This article draws on argument-based validation to gather and evaluate construct-related evidence (i.e., the explanation inference) of a high-stakes test. The data stemmed from the listening component of a French test used for immigration to Canada through the province of Quebec. An expert panel with varied backgrounds in applied linguistics reviewed and associated the items of two operational test forms to four listening comprehension sub-skills identified in selected sources of second language listening theory. Based on the expert panel recommendations, two confirmatory factor models were fit to examinees’ response data. The models fit the data well, providing backing for the explanation inference but suggesting construct under-representation for one of the test forms examined. The argument-based approach to validation yielded principled guidelines to evaluate construct coverage of the test across forms, providing insightful guidance on how to organize construct evidence from an argumentation perspective. Implications are discussed as they relate to the operationalization of argument-based validation in high-stakes settings.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.049 | 0.197 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.004 | 0.001 |
| Science and technology studies | 0.002 | 0.002 |
| Scholarly communication | 0.007 | 0.003 |
| Open science | 0.002 | 0.003 |
| Research integrity | 0.003 | 0.002 |
| Insufficient payload (model declined to judge) | 0.007 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".