Reliability and Sensitivity of a Virtual Assessment Developed for Workplace Concussions: Protocol for a Method-Comparison Study
Bibliographic record
Abstract
BACKGROUND: Workplace mild traumatic brain injuries are frequently associated with persistent symptoms, leading to a reduction in productivity at work or even disability. People who sustain workplace injuries frequently need rehabilitation and support, and the challenges of delivering these services was heightened during the COVID-19 pandemic as injured workers had to be cared for remotely. Currently, clinicians are conducting both in-person and virtual (remote) concussion assessments; however, the measures that are being used to complete these assessments have undocumented psychometric properties. OBJECTIVE: This study will document the psychometric properties of the clinical measures that are being used remotely and their ability to produce similar results to in-person assessments. Specifically, through this method-comparison study, we aim to (1) evaluate the sensitivity of the measures included in a virtual assessment toolkit when compared to an in-person assessment and (2) determine the interrater and intrarater reliabilities of the measures included in a virtual assessment toolkit. METHODS: Patient participants (people living with acquired brain injuries) will attend two assessments (in person and virtual) at the Ottawa Hospital. The two assessments will be identical, consisting of the measures included in our previously developed virtual concussion assessment toolkit, which includes finger-to-nose testing, the Vestibular/Ocular Motor Screening tool, balance testing, cervical spine range of motion, saccades testing, and evaluation of effort. All virtual assessments will occur using the Microsoft Teams platform and will be audio/video-recorded. The clinician assessor and patient participant will complete a feedback form following completion of the assessments. A different clinician will also document the findings on observed videos of the virtual assessment shortly after completion of both in-person and virtual assessments and approximately 1 month later. Interrater reliability will be assessed by comparing the second clinician's observation with the first clinician's initial virtual assessment. Intrarater reliability will be evaluated by comparing the second clinician's observation with their own assessment approximately 1 month later. Sensitivity will be documented by comparing the findings (identification of abnormality) of the in-person assessment completed by the initial clinician assessor with those of the second clinician assessor on the observation of the recording of the virtual assessment. RESULTS: As of May 2024, we have recruited 7 clinician assessors and completed study assessments with 39 patient participants. The study recruitment is expected to be completed by September 2024. CONCLUSIONS: Currently, it is unknown if completing concussion assessments virtually produces similar results to the in-person assessment. This work will serve as a first step to determining the similarity of the virtual assessment to the matching in-person assessment and will provide information on the reliability of the virtual assessment. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/57663.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.148 | 0.168 |
| Meta-epidemiology (narrow) | 0.005 | 0.003 |
| Meta-epidemiology (broad) | 0.006 | 0.006 |
| Bibliometrics | 0.004 | 0.004 |
| Science and technology studies | 0.004 | 0.004 |
| Scholarly communication | 0.003 | 0.003 |
| Open science | 0.004 | 0.003 |
| Research integrity | 0.005 | 0.005 |
| Insufficient payload (model declined to judge) | 0.023 | 0.006 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".