Quality comparison of remote vs. in‐person digital speech assessment for dementia
Bibliographic record
Abstract
Abstract Background The COVID‐19 pandemic has brought the need for reliable, remote assessments for clinical trials into sharp focus. Remote, home‐based assessment reduces patient and caregiver burden, enabling more frequent monitoring. There are concerns, however, regarding the quality of in‐clinic vs. remote assessments. In the present study, we compared the quality of verbal responses to a tablet‐based speech assessment in patients with dementia across two settings: in‐person at a clinical research site, and remotely, at home. Method The in‐person sample consisted of individuals with Alzheimer’s disease, participating in a clinical trial. The remote sample consisted of individuals with variants of Frontotemporal dementia, participating in a longitudinal observational study. In‐person assessments were conducted at a clinical research site and administered by a trained rater. Remote assessments were conducted in the participant’s home, by a caregiver who had received assessment training and a tablet by mail. 575 in‐person speech samples and 574 remote speech samples were compared. All samples were manually transcribed and tagged for recording anomalies or task compliance issues by trained transcriptionists. Result Overall incidence of recording anomalies was low and did not differ significantly (p = 0.63) between in‐person (10.1%; 58/574) and remote (11.1%, 64/575) recordings. Less than 1% of samples in either study were marked as “incomplete task”, “low audio quality” or “noisy background”. Incidences of clinician/caregiver interference were more frequent for remote recordings (1.2% in‐person, 4.5% remote, p = 0.001) and incidences of quiet participants were more frequent for in‐person recordings (4.5% in‐person, 1.0% remote, p < 0.001), though both were relatively rare overall. The mean duration of in‐person samples was significantly longer than remote samples, but mean speech rate did not differ. Conclusion This study suggests that remote speech assessments yield recordings of comparable quality to in‐person assessments. We found higher, though still low, rates of caregiver interference for remote assessment, which should be monitored and mitigated in future remote assessment. Remote assessments yielded shorter recordings, but this may be due to the different dementia diagnoses across groups. Future work should compare the same participants across both assessment settings.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.017 | 0.053 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.002 | 0.001 |
| Science and technology studies | 0.000 | 0.001 |
| Scholarly communication | 0.001 | 0.001 |
| Open science | 0.001 | 0.001 |
| Research integrity | 0.001 | 0.000 |
| Insufficient payload (model declined to judge) | 0.003 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".