Standardizing assessment of spoken discourse in aphasia: A working group with deliverables
Bibliographic record
Abstract
Purpose: The heterogeneous nature of measures, methods, and analyses reported in the aphasia spoken discourse literature precludes comparison of outcomes across studies (e.g., meta-analyses) and inhibits replication. Furthermore, funding and time constraints significantly hinder collecting test-retest data on spoken discourse outcomes. This research note describes the development and structure of a working group, designed to address major gaps in the spoken discourse aphasia literature, including a lack of standardization in methodology, analysis, and reporting, as well as nominal data regarding the psychometric properties of spoken discourse outcomes.Method: The initial initiatives for this working group are to: 1) propose recommendations regarding standardization of spoken discourse collection, analysis, and reporting in aphasia, based on the results of an international survey and a systematic literature review; and, 2) create a database of test-retest spoken discourse data from individuals with and without aphasia. The survey of spoken discourse collection, analysis, and interpretation procedures was distributed to clinicians and researchers involved in aphasia assessment and rehabilitation September – November 2019. We will publish survey results and recommend standards for collecting, analyzing, and reporting spoken discourse in aphasia. A multisite endeavor to collect test-retest spoken discourse data from individuals with and without aphasia will be initiated. This test-retest information will be contributed to a central site for transcription and analysis, and data will be subsequently openly curated.Conclusion: The goal of the working group is to create recommendations for field-wide standards in methods, analysis and reporting of spoken discourse outcomes, as has been done across other related disciplines (e.g., CONSORT, EQUATOR, COBIDAS). Additionally, the creation of a database through our multi-site collaboration will allow the identification of psychometrically sound outcome measures and norms that can be used by clinicians and researchers to assess spoken discourse abilities in aphasia.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.485 | 0.475 |
| Meta-epidemiology (narrow) | 0.004 | 0.003 |
| Meta-epidemiology (broad) | 0.003 | 0.007 |
| Bibliometrics | 0.008 | 0.003 |
| Science and technology studies | 0.007 | 0.005 |
| Scholarly communication | 0.011 | 0.015 |
| Open science | 0.012 | 0.036 |
| Research integrity | 0.007 | 0.012 |
| Insufficient payload (model declined to judge) | 0.006 | 0.006 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".