Rater Behaviour When Judging Language Learners’ Pragmatic Appropriateness in Extended Discourse
Bibliographic record
Abstract
The criteria by which raters judge pragmatic appropriateness of language learners’ speech acts are underexamined, especially when raters evaluate extended discourse. To shed more light on this process, the present study investigated what factors are salient to raters when scoring pragmatic appropriateness of extended request sequences, and which specific aspects of performance they attend to as appropriate or inappropriate. Three judges evaluated request sequences using a 6-point scale, marked appropriate and inappropriate elements of each request, and explained how they approached the rating of each response. It was found that all raters oriented to the appropriateness of a request sequence as a whole, paying attention not only to the request proper but also to all follow-up moves, including appreciation and closing. Additionally, raters oriented to the surrounding context: the same expressions, such as a specific appreciation statement, were rated as appropriate in some contexts and inappropriate in others. Raters also oriented to pragmatic competence broadly, paying attention not only to appropriate pragmatic strategies and expressions in a particular context, but also to such aspects as intonation and cultural knowledge. Finally, while native and near-native speaker tendencies were observed, target speaker norms were not. Implications for pragmatics teaching and assessment are discussed.Les critères selon lesquels les évaluateurs jugent la pertinence pragmatique des actes de langage d’apprenants de langue n’ont pas suffisamment fait l’objet d’études, notamment lors de l’évaluation de longues conversations. Pour éclairer davantage le processus, la présente étude a cherché à déterminer quels facteurs les évaluateurs jugent importants dans la pertinence pragmatique de séquences de requête étendues, et quels aspects spécifiques de la performance ils estiment appropriés ou pas. Trois juges ont évalué des séquences de requête selon une échelle de 6 points, ont indiqué les éléments appropriés et inappropriés de chaque requête et ont expliqué comment ils avaient abordé l’évaluation de chaque réponse. Les résultats indiquent que tous les évaluateurs jugeaient de la pertinence d’une séquence de requête dans son intégralité, portant attention non seulement à la requête comme telle mais aussi à toutes les démarches qui la suivaient, y compris le remerciement et la clôture. De plus, les évaluateurs tenaient compte du contexte : ils jugeaient qu’une même expression, une déclaration spécifique d’appréciation par exemple, était appropriée dans un contexte donné alors qu’elle ne l’était pas dans un autre. Ils ont également considéré la compétence pragmatique globale, notant, au delà des stratégies et des expressions pragmatiques appropriées dans un contexte donné, des aspects comme l’intonation et les connaissances culturelles. Finalement, si les évaluateurs ont observé des tendances de locuteurs natifs ou quasi-natifs, on ne peut en dire autant des normes de la langue cible. On discute des incidences de l’étude sur l’enseignement et l’évaluation des compétences pragmatiques.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.107 | 0.265 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.002 | 0.001 |
| Science and technology studies | 0.001 | 0.002 |
| Scholarly communication | 0.002 | 0.002 |
| Open science | 0.001 | 0.002 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.001 | 0.002 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".