Opening the Black Box of Electronic Health: Collecting, Analyzing, and Interpreting Log Data
Bibliographic record
Abstract
In electronic health (eHealth) research, limited insight has been obtained on process outcomes or how the use of technology has contributed to the users' ability to have a healthier life, improved well-being, or activate new attitudes in their daily tasks. As a result, eHealth is often perceived as a black box. To open this black box of eHealth, methodologies must extend beyond the classic effect evaluations. The analyses of log data (anonymous records of real-time actions performed by each user) can provide continuous and objective insights into the actual usage of the technology. However, the possibilities of log data in eHealth research have not been exploited to their fullest extent. The aim of this paper is to describe how log data can be used to improve the evaluation and understand the use of eHealth technology with a broader approach than only descriptive statistics. This paper serves as a starting point for using log data analysis in eHealth research. Here, we describe what log data is and provide an overview of research questions to evaluate the system, the context, the users of a technology, as well as the underpinning theoretical constructs. We also explain the requirements for log data, the starting points for the data preparation, and methods for data collection. Finally, we describe methods for data analysis and draw a conclusion regarding the importance of the results for both scientific and practical applications. The analysis of log data can be of great value for opening the black box of eHealth. A deliberate log data analysis can give new insights into how the usage of the technology contributes to found effects and can thereby help to improve the persuasiveness and effectiveness of eHealth technology and the underpinning behavioral models.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.107 | 0.316 |
| Meta-epidemiology (narrow) | 0.002 | 0.002 |
| Meta-epidemiology (broad) | 0.002 | 0.001 |
| Bibliometrics | 0.009 | 0.010 |
| Science and technology studies | 0.004 | 0.008 |
| Scholarly communication | 0.014 | 0.020 |
| Open science | 0.003 | 0.007 |
| Research integrity | 0.003 | 0.005 |
| Insufficient payload (model declined to judge) | 0.004 | 0.003 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".