Measuring Engagement in Provider-Guided Digital Health Interventions With a Conceptual and Analytical Framework Using Nurse WRITE as an Exemplar: Exploratory Study With an Iterative Approach
Bibliographic record
Abstract
BACKGROUND: Limited guidance exists for analyzing participant engagement in provider-guided digital health interventions (DHIs). System usage is commonly assessed, with acknowledged limitations in measuring socio-affective and cognitive aspects of engagement. Nurse WRITE, an 8-week web-based nurse-guided DHI for managing symptoms among women with recurrent ovarian cancer, offers an opportunity to develop a framework for assessing multidimensional engagement. OBJECTIVE: This study aims to develop a conceptual and analytic framework to measure socio-affective, cognitive, and behavioral engagement with provider-guided DHIs. We then illustrate the framework's ability to describe and categorize engagement using Nurse WRITE as an example. METHODS: A sample of 68 participants from Nurse WRITE who posted on the message boards were included. We adapted a prior framework for conceptualizing and operationalizing engagement across 3 dimensions and finalized a set of 6 distinct measures. Using patients' posts, we created 2 socio-affective engagement measures-total count of socio-affective engagement classes (eg, sharing personal experience) and total word count-and 2 cognitive engagement measures-total count of cognitive engagement classes (eg, asking information-seeking questions) and average question completion percentage. Additionally, we devised behavioral engagement measures using website data-the total count of symptom care plans and plan reviews. k-Means clustering categorized the participants into distinct groups based on levels of engagement across 3 dimensions. Descriptive statistics and narratives were used to describe engagement in 3 dimensions. RESULTS: On average, participants displayed socio-affective engagement 34.7 times, writing 14,851 words. They showed cognitive engagement 19.4 times, with an average of 78.3% completion of nurses' inquiries. Participants also submitted an average of 1.6 symptom care plans and 0.7 plan reviews. Participants were clustered into high (n=13), moderate (n=17), and low engagers (n=38) based on the 6 measures. High engagers wrote a median of 36,956 (IQR 26,199-46,265) words. They demonstrated socio-affective engagement approximately 81 times and cognitive engagement around 46 times, approximately 6 times that of the low engagers and twice that of the moderate engagers. High engagers had a median of 91.7% (IQR 82.2%-93.7%) completion of the nurses' queries, whereas moderate engagers had 86.4% (IQR 80%-96.4%), and low engagers had 68.3% (IQR 60.1%-79.6%). High engagers completed a median of 3 symptom care plans and 2 reviews, while moderate engagers completed 2 plans and 1 review. Low engagers completed a median of 1 plan with no reviews. CONCLUSIONS: This study developed and reported an engagement framework to guide behavioral intervention scientists in understanding and analyzing participants' engagement with provider-guided DHIs. Significant variations in engagement levels across 3 dimensions highlight the importance of measuring engagement with provider-guided DHIs in socio-affective, cognitive, and behavioral dimensions. Future studies should validate the framework with other DHIs, explore the influence of patient and provider factors on engagement, and investigate how engagement influences intervention efficacy.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.024 | 0.027 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.002 | 0.001 |
| Science and technology studies | 0.003 | 0.004 |
| Scholarly communication | 0.003 | 0.003 |
| Open science | 0.002 | 0.005 |
| Research integrity | 0.001 | 0.002 |
| Insufficient payload (model declined to judge) | 0.001 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".