Actively managed products: Think-aloud data and methods in applied linguistics research
Bibliographic record
Abstract
Abstract Verbal reports, specifically in the form of concurrent verbalizations (i.e., think-alouds [TAs]), have played a foundational role in the production of knowledge in applied linguistics. Most often drawn upon because the talk they generate is deemed to accurately reflect individual learners’ thought or cognitive processes as they complete an L2 task, concurrent verbalization methods have been central to investigations of and claims about the learning, use, and assessment of L2 vocabulary, listening, speaking, reading, and writing (among others). And although critical discussion concerning the quality of spoken data obtained through concurrent verbalization methods continues among L2 researchers (e.g., Cohen, Andrew D. 1987. Using verbal reports in research on language learning. In Claus Færch & Gabriele Kasper (eds.),Introspection in second language research, 82–95. Philadelphia: Multilingual Matters; Cohen, Andrew D. 1996. Verbal reports as a source of insights into second language learner strategies.Applied Language Learning7(1–2). 5–24; Cohen, Andrew D. 2013. Verbal report. In Carol A. Chapelle (ed.),The encyclopedia of applied linguistics. Oxford: Wiley-Blackwell), the majority of this discussion has focused primarily on how best to generate talk which “more accurately reflect[s] the actual thought processes” of L2 users (Cohen, Andrew D. 2013. Verbal report. In Carol A. Chapelle (ed.),The encyclopedia of applied linguistics. Oxford: Wiley-Blackwell: 1). The result has been to further naturalize approaches to concurrent verbalizations which treat language as a neutral means for accessing cognition, and similarly, which treat the verbalizations themselves as individually accomplished events. In this article, my aim is to diversify the critical discussion by describing how discursive psychology (e.g., Edwards, Derek & Jonathan Potter. 1992.Discursive psychology. New York: Sage; Potter, Jonathan. 2006. Cognition and conversation.Discourse Studies8(1). 131–140) and a conversation analytic perspective (e.g., Kasper, Gabriele. 2009. Locating cognition in second language interaction and learning: Inside the skull or in public view?International Review of Applied Linguistics47. 11–36; Markee, Numa & Mi-Suk Seo. 2009. Learning talk analysis.International Review of Applied Linguistics47. 37–63) can be combined to present an alternative to both ‘naturalized’, as well as sociocultural, understandings of concurrent verbalization data and methods. To this end, after establishing some of the key differences between information processing, sociocultural, and discursive approaches, I draw on data from two recently published TA-based studies in an attempt to accomplish two goals: the first is to shift critical discussion towards issues of epistemology, methodology, and research representation, and the second is to identify methodological issues about which researchers working from different conceptual orientations might engage in cross-paradigmatic dialogue.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.139 | 0.225 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.008 | 0.008 |
| Science and technology studies | 0.003 | 0.007 |
| Scholarly communication | 0.010 | 0.007 |
| Open science | 0.003 | 0.007 |
| Research integrity | 0.002 | 0.004 |
| Insufficient payload (model declined to judge) | 0.006 | 0.003 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".