The Dutch COVID-19 Notification App: Lessons Learned From a Mixed Methods Evaluation Among End Users and Contact-Tracing Employees
Bibliographic record
Abstract
BACKGROUND: The Dutch CoronaMelder (CM) app is the official Dutch contact-tracing app (CTA). It has been used to contain the spread of the SARS-CoV-2 in the Netherlands. It allows its users and those of connected apps to anonymously exchange warnings about potentially high-risk contacts with individuals infected with the SARS-CoV-2. OBJECTIVE: The goal of this mixed methods study is to understand the use of CTA in the pandemic and its integration into the Municipal Health Services (MHS) efforts of containment through contact tracing. Moreover, the study aims to investigate both the motivations and user experience-related factors concerning adherence to quarantine and isolation measures. METHODS: A topic analysis of 56 emails and a web-based survey of 1937 adults from the Netherlands, combined with a series of 48 in-depth interviews with end users of the app and 14 employees of the Dutch MHS involved in contact tracing, were conducted. Mirroring sessions were held (n=2) with representatives from the development (n=2) and communication teams (n=2) responsible for the creation and implementation of the CM app. RESULTS: Topic analysis and interviews identified procedural and technical issues in the use of the CTA. Procedural issues included the lack of training of MHS employees in the use of CTAs. Technical issues identified for the end users included the inability to send notifications without phone contact with the MHS, unwarranted notifications, and nightly notifications. Together, these issues undermined confidence in and satisfaction with the app's use. The interviews offered a deeper understanding of the various factors at play and their effects on users; for example, the mixed experiences of the app's users, the end user's own fears, and uncertainties concerning the SARS-CoV-2; problematic infrastructure at the time of the app's implementation on the side of the health services; the effects of the society-wide efforts in containment of the SARS-CoV-2 on the CM app's perception, resulting in further doubts concerning the app's effectiveness among MHS workers and citizens; and problems with adherence to behavioral measures propagated by the app because of the lack of confidence in the app and uncertainty concerning the execution of the behavioral measures. All findings were evaluated with the app's creators and have since contributed to improvements. CONCLUSIONS: Although most participants perceived the app positively, procedural and technical issues identified in this study limited satisfaction and confidence in the CM app and affected its adoption and long-term use. Moreover, these same issues negatively affected the CM app's effectiveness in improving compliance with behavioral measures aimed at reducing the spread of the SARS-CoV-2. This study offers lessons learned for future eHealth interventions in pandemics. Lessons that can aid in more effective design, implementation, and communication for more effective and readily adoptable eHealth applications.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.181 | 0.150 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.002 | 0.002 |
| Bibliometrics | 0.001 | 0.001 |
| Science and technology studies | 0.004 | 0.002 |
| Scholarly communication | 0.007 | 0.006 |
| Open science | 0.005 | 0.008 |
| Research integrity | 0.003 | 0.003 |
| Insufficient payload (model declined to judge) | 0.003 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".