Stakeholder voices: Validity argument for score meaning and use of the Test of English for International Communication (TOEIC) in international business workplaces in South Korea
Bibliographic record
Abstract
Competent English communication skills are essential to success within international business workplaces. To assess employees’ qualifications for international businesses, international companies have used the Test of English for International Communication (TOEIC) over the past 30 years in South Korea, which houses the largest TOEIC test taker population worldwide. However, some researchers argue that the TOEIC, designed by Educational Testing Service (ETS), may not fully reflect the use of English in international business contexts. In such contexts, the focus is interactive communication skills, with cognizance of different English accents and business partners’ cultures. Limited empirical studies have investigated the extent to which TOEIC scores represent English constructs and how much the TOEIC achieves its designated purposes in international business workplaces. Drawing on an argument-based validation framework using the stakeholder approach, the perceived constructs of English in international business workplaces and the extent to which the TOEIC’s designated purposes were achieved were investigated by collecting various validity-related evidence from multiple stakeholders, i.e., ETS; YBM, the TOEIC administrator; and employers and employees in international business workplaces. Multiple methods (i.e., documents from ETS and YBM, interviews with six employers and ten employees, and online questionnaires with 281 employees in international business companies in Korea) were used to juxtapose the findings from each stakeholder group. The findings from ETS’s documents framed baseline validity evidence to compare the findings from YBM’s documents and interview/questionnaire data that supported and challenged the TOEIC’s intended interpretations and uses. The overall findings represent the complexity of relationships within English constructs in international business workplaces and between stakeholders’ understanding and uses of TOEIC scores that were affected by contextual factors in South Korea, by highlighting the why and how. The findings can contribute to re-operationalizing the TOEIC’s constructs and call for including contextual factors in validation frameworks and studies. In particular, investigations of contextual factors cannot be overlooked to affect beneficial consequences in score use contexts. ETS and validation researchers may need to pay more attention to contextual factors and collect validity evidence in collaboration with relevant stakeholders.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.297 | 0.540 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.006 | 0.003 |
| Science and technology studies | 0.013 | 0.031 |
| Scholarly communication | 0.009 | 0.016 |
| Open science | 0.004 | 0.021 |
| Research integrity | 0.006 | 0.007 |
| Insufficient payload (model declined to judge) | 0.002 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".