Rater Behaviour When Judging Language Learners’ Pragmatic Appropriateness in Extended Discourse
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
The criteria by which raters judge pragmatic appropriateness of language learners’ speech acts are underexamined, especially when raters evaluate extended discourse. To shed more light on this process, the present study investigated what factors are salient to raters when scoring pragmatic appropriateness of extended request sequences, and which specific aspects of performance they attend to as appropriate or inappropriate. Three judges evaluated request sequences using a 6-point scale, marked appropriate and inappropriate elements of each request, and explained how they approached the rating of each response. It was found that all raters oriented to the appropriateness of a request sequence as a whole, paying attention not only to the request proper but also to all follow-up moves, including appreciation and closing. Additionally, raters oriented to the surrounding context: the same expressions, such as a specific appreciation statement, were rated as appropriate in some contexts and inappropriate in others. Raters also oriented to pragmatic competence broadly, paying attention not only to appropriate pragmatic strategies and expressions in a particular context, but also to such aspects as intonation and cultural knowledge. Finally, while native and near-native speaker tendencies were observed, target speaker norms were not. Implications for pragmatics teaching and assessment are discussed.Les critères selon lesquels les évaluateurs jugent la pertinence pragmatique des actes de langage d’apprenants de langue n’ont pas suffisamment fait l’objet d’études, notamment lors de l’évaluation de longues conversations. Pour éclairer davantage le processus, la présente étude a cherché à déterminer quels facteurs les évaluateurs jugent importants dans la pertinence pragmatique de séquences de requête étendues, et quels aspects spécifiques de la performance ils estiment appropriés ou pas. Trois juges ont évalué des séquences de requête selon une échelle de 6 points, ont indiqué les éléments appropriés et inappropriés de chaque requête et ont expliqué comment ils avaient abordé l’évaluation de chaque réponse. Les résultats indiquent que tous les évaluateurs jugeaient de la pertinence d’une séquence de requête dans son intégralité, portant attention non seulement à la requête comme telle mais aussi à toutes les démarches qui la suivaient, y compris le remerciement et la clôture. De plus, les évaluateurs tenaient compte du contexte : ils jugeaient qu’une même expression, une déclaration spécifique d’appréciation par exemple, était appropriée dans un contexte donné alors qu’elle ne l’était pas dans un autre. Ils ont également considéré la compétence pragmatique globale, notant, au delà des stratégies et des expressions pragmatiques appropriées dans un contexte donné, des aspects comme l’intonation et les connaissances culturelles. Finalement, si les évaluateurs ont observé des tendances de locuteurs natifs ou quasi-natifs, on ne peut en dire autant des normes de la langue cible. On discute des incidences de l’étude sur l’enseignement et l’évaluation des compétences pragmatiques.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.001 | 0.001 |
| Open science | 0.001 | 0.000 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.009 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it