Usability of a Co-designed eHealth Prototype for Caregivers: Combination Study of Three Frameworks
Bibliographic record
Abstract
BACKGROUND: Co-design (or the participation of users) has shown great potential in the eHealth domain, demonstrating positive results. Nevertheless, the co-design approach cannot guarantee the usability of the system designed, and usability assessment is a complex analysis to perform, as evaluation criteria will differ depending on the usability framework (or set of criteria) used. ISO (International Organization for Standardization) on usability (ISO 9241-210), Nielsen heuristic, and Garrett element of user experience inform different yet complementary aspects of usability. OBJECTIVE: This study aims to assess the usability and user experience of a co-design prototype by combining 3 complementary frameworks. METHODS: To help caregivers provide care for functionally impaired older people, an eHealth tool was co-designed with caregivers, health and social service professionals, and community workers assisting caregivers. The prototype was a website that aims to support the help-seeking process for caregivers (finding resources) and allow service providers to advertise their services (offering resources). We chose an exploratory study method to assess usability in terms of each objective. The first step was to assess users' first impressions of the website. The second was a task scenario with a think-aloud protocol. The final step was a semistructured interview. All steps were performed individually (with a moderator) in a single session. The data were analyzed using 3 frameworks. RESULTS: A total of 10 participants were recruited, 5 for each objective of the website. We were able to identify several usability problems, most of which were located in the information design and interface design dimensions (Garrett framework). Problems in both dimensions were mainly coded as effectiveness and efficiency (ISO framework) and error prevention and match between the systemand the real world (Nielsen heuristic). CONCLUSIONS: Our study provided a novel contribution about usability analysis by combining the 3 different models to classify the problems found. This combination provided a holistic understanding of the usability improvements needed. It can also be used to analyze other eHealth products. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): RR2-10.2196/11634.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.018 | 0.048 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.002 |
| Bibliometrics | 0.003 | 0.001 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.002 | 0.003 |
| Open science | 0.001 | 0.002 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.001 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".