A Tool That Assesses the Evidence, Transparency, and Usability of Online Health Information: Development and Reliability Assessment
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
BACKGROUND: The internet is commonly used by older adults to obtain health information and this trend has markedly increased in the past decade. However, studies illustrate that much of the available online health information is not informed by good quality evidence, developed in a transparent way, or easy to use. Furthermore, studies highlight that the general public lacks the skills necessary to distinguish between online products that are credible and trustworthy and those that are not. A number of tools have been developed to assess the evidence, transparency, and usability of online health information; however, many have not been assessed for reliability or ease of use. OBJECTIVE: The first objective of this study was to determine if a tool assessing the evidence, transparency, and usability of online health information exists that is easy and quick to use and has good reliability. No such tool was identified, so the second objective was to develop such a tool and assess it for reliability when used to assess online health information on topics of relevant to optimal aging. METHODS: An electronic database search was conducted between 2002 and 2012 to identify published papers describing tools that assessed the evidence, transparency, and usability of online health information. Papers were retained if the tool described was assessed for reliability, assessed the quality of evidence used to create online health information, and was quick and easy to use. When no one tool met expectations, a new instrument was developed and tested for reliability. Reliability between two raters was assessed using the intraclass correlation coefficient (ICC) for each item at two time points. SPSS Statistics 22 software was used for statistical analyses and a one-way random effects model was used to report the results. The overall ICC was assessed for the instrument as a whole in July 2015. The threshold for retaining items was ICC>0.60 (ie, "good" reliability). RESULTS: All tools identified that evaluated online health information were either too complex, took a long time to complete, had poor reliability, or had not undergone reliability assessment. A new instrument was developed and assessed for reliability in April 2014. Three items had an ICC<0.60 (ie, "good" reliability). One of these items was removed ("minimal scrolling") and two were retained but reworded for clarity. Four new items were added that assessed the level of research evidence that informed the online health information and the tool was retested in July 2015. The total ICC score showed excellent agreement with both single measures (ICC=0.988; CI 0.982-0.992) and average measures (ICC=0.994; CI 0.991-0.996). CONCLUSIONS: The results of this study suggest that this new tool is reliable for assessing the evidence, transparency, and usability of online health information that is relevant to optimal aging.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.006 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.001 | 0.000 |
| Scholarly communication | 0.000 | 0.002 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it