The Impact of Electronic Administration of the MMSE on Error Rates and Data Quality
Bibliographic record
Abstract
Background: The Mini-Mental State Examination (MMSE) is a widely used measure of cognition in clinical practice and is a frequently used screening and outcome measure for cognition-enhancing clinical drug trials for Alzheimer’s Disease. Scoring and administration errors on the MMSE in a clinical drug trial can contribute to placebo response and unreliable data. Examples of such errors range from basic addition errors to failure to adhere to structured administration instructions. Prior research from independent clinical review of paper versions of the scale as part of a data quality monitoring program in worldwide studies has demonstrated that MMSE error rates among clinical trial raters tend to be upwards of 30% on initial administration. An enhanced tablet-based electronic (eCOA) version of the MMSE was developed for clinical trials by Bracket and has been utilized in worldwide clinical trials since 2013. The scale is administered in a system compliant with US Food and Drug Administration guidelines on the use of electronic medical records in trials and was designed to be equivalent to the paper MMSE. The eCOA version was augmented with internal logic, automated scoring and addition, standardized instructions, and enforcement of administration conventions. These enhancements were intended to decrease errors and increase the likelihood that the scale would be consistently administered and scored according to standard conventions. Objective: Our objectives were to (1) determine if error rates using the eCOA MMSE were reduced when compared to standard paper administration and comparable to eCOA error rates from an analysis published in 2015 and (2) consider the impact of utilizing enhanced eCOA versions of scales like the MMSE in clinical practice. Methods: All data from this analysis and comparative analyses were from multinational studies that trained and certified raters on the MMSE and included an in-study data monitoring program of clinical reviews to detect scoring and administration errors. Error rates on the enhanced electronic versions of the scales used in a late-phase Alzheimer’s study were compared to those previously seen in other studies when using paper versions and the enhanced eCOA MMSE. The dataset for this eCOA analysis was limited to English-speaking sites. Results: A total of 66/626 (10.5%) of the enhanced eCOA MMSEs reviewed at the inclusion visits contained an error in scoring which required outreach to the rater. This represents a statistically significant reduction (P<.0001) in the error rates compared to paper administration of the MMSE (877/2889, 30.4%) and is consistent with the findings from a prior analysis of an eCOA MMSE dataset from another trial (107/990,10.8%). Conclusions: Use of enhanced eCOA for the MMSE in clinical trials has significantly improved error rates and data quality compared to standard paper administration. Enhanced eCOA that is designed to minimize errors and enforce administration standards could be a useful tool to reduce common errors and increase the precision of the MMSE use in clinical practice.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.054 | 0.033 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.002 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; both teacher heads agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".