Co-Designing a Web-Based and Tablet App to Evaluate Clinical Outcomes of Early Psychosis Service Users in a Learning Health Care Network: User-Centered Design Workshop and Pilot Study
Bibliographic record
Abstract
BACKGROUND: The Early Psychosis Intervention Network of California project, a learning health care network of California early psychosis intervention (EPI) programs, prioritized incorporation of community partner feedback while designing its eHealth app, Beehive. Though eHealth apps can support learning health care network data collection aims, low user acceptance or adoption can pose barriers to successful implementation. Adopting user-centered design (UCD) approaches, such as incorporation of user feedback, prototyping, iterative design, and continuous evaluation, can mitigate these potential barriers. OBJECTIVE: We aimed to use UCD during development of a data collection and data visualization web-based and tablet app, Beehive, to promote engagement with Beehive as part of standard EPI care across a diverse user-base. METHODS: Our UCD approach included incorporation of user feedback, prototyping, iterative design, and continuous evaluation. This started with user journey mapping to create storyboards, which were then presented in UCD workshops with service users, their support persons, and EPI providers. We incorporated feedback from these workshops into the alpha version of Beehive, which was also presented in a UCD workshop. Feedback was again incorporated into the beta version of Beehive. We provided Beehive training to 4 EPI programs who then piloted Beehive's beta version. During piloting, service users, their support persons, and EPI program providers completed Beehive surveys at enrollment and every 6 months after treatment initiation. To examine preliminary user acceptance and adoption during the piloting phase, we assessed rates of participant enrollment and survey completion, with a particular focus on completion of a prioritized survey: the Modified Colorado Symptom Index. RESULTS: UCD workshop feedback resulted in the creation of new workflows and interface changes in Beehive to improve the user experience. During piloting, 48 service users, 42 support persons, and 72 EPI program providers enrolled in Beehive. Data were available for 88% (n=42) of service users, including self-reported data for 79% (n=38), collateral-reported data for 42% (n=20), and clinician-entered data for 17% (n=8). The Modified Colorado Symptom Index was completed by 54% (n=26) of service users (total score: mean 24.16, SD 16.81). In addition, 35 service users had a support person who could complete the Modified Colorado Symptom Index, and 56% (n=19) of support persons completed it (mean 26.71, SD 14.43). CONCLUSIONS: Implementing UCD principles while developing the Beehive app resulted in early workflow changes and produced an app that was acceptable and feasible for collection of self-reported clinical outcomes data from service users. Additional support is needed to increase collateral-reported and clinician-entered data.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.022 | 0.027 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.001 | 0.000 |
| Science and technology studies | 0.002 | 0.001 |
| Scholarly communication | 0.003 | 0.002 |
| Open science | 0.002 | 0.002 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.003 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".