Designing public web information systems with quality in mind
Bibliographic record
Abstract
Purpose This paper aims to describe evolution of a new public information web site, through evaluation‐refinement prototyping cycles. Design/methodology/approach An expanding range of participants is being engaged in formal evaluations as the site design evolves. The Flesch‐Kincaid Grade Level Score is applied to assess ease of reading in the wording used; the National Quality Forum guideline statements are applied to determine whether the prototyping design process is meeting performance expectations; and then Nielsen's heuristics are applied to evaluate ease of use of the latest prototype. Findings The page wordings started at a high reading grade level to be technically correct, with a strategy to progressively reduce levels without losing meaning. Reading level was reduced to an average of two and as much as six grades through editing between the third and fourth‐generation prototypes. None of the National Quality Forum principles were found missing from the development process. The prototype web site was ranked at the middle compared to official public web sites of seven other States' healthcare‐associated infection programs, some of which had been open to the public for more than a year. Many of the heuristic violations that weighed against the prototype were described as being minor and easily fixed. Collaboration between a State health department and a university to advance this evaluation‐refinement process was valuable to both parties, enhancing the ability to produce a new public information web site that is more likely to meet the needs of its intended audience. Practical implications In response to increasing expectations of transparency and accountability, a growing number of public web sites are displaying hospital performance data. Washington State's mandatory public reporting of healthcare‐associated infection rates is a recent example of this trend. The Department of Health is required by law to launch a public information web site by December 2009. The research was based on an evidence‐based approach to understand and meet the information needs of the public. Originality/value Although few studies have evaluated the usage and impact of hospital comparison web sites, these studies uniformly show relatively low usage and disappointing impact. Using the research literature, issues thought to account for poor usage and low impact, and developed design principles that address this poor past performance were identified. Throughout 2008 and 2009, successive prototypes were developed for the web site structure guided by those principles and refined each generation of prototype through focus group evaluations. This paper explains the approach, and summarizes results from the evaluations, leading to improvements before the final design first opens to the general public.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.033 | 0.056 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.002 | 0.002 |
| Science and technology studies | 0.002 | 0.003 |
| Scholarly communication | 0.013 | 0.013 |
| Open science | 0.003 | 0.005 |
| Research integrity | 0.002 | 0.002 |
| Insufficient payload (model declined to judge) | 0.004 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".