Development of a survey to support assessment of safety learning systems
Bibliographic record
Abstract
BACKGROUND: Patient safety learning systems play a critical role in supporting safety culture in healthcare organisations. A lack of explicit standards leads to inconsistent implementation across organisations, causing uncertainty about their roles and impact. Organisations can address inconsistent implementation by using a self-assessment tool based on agreed-on best practices. Therefore, we aimed to create a survey instrument to assess an organisation's approach to learning from safety events. METHODS: The foundation for this work was a recent systematic review that defined features associated with the performance of a safety learning system. We organised features into themes and rephrased them into questions (items). Face validity was checked, which included independent pre-testing to ensure comprehensibility and parsimony. It also included clinical sensibility testing in which a representative sample of leaders in quality at a large teaching hospital (The Ottawa Hospital) answered two questions to judge each item for clarity and necessity. If more than 20% of respondents judged a question unclear or unnecessary, we modified or removed that question accordingly. Finally, we checked the internal consistency of the questionnaire using Cronbach's alpha. RESULTS: We initially developed a 47-item questionnaire based on a prior systematic review. Pre-testing resulted in the modification of 15 of the questions, 2 were removed and 2 questions were added to ensure comprehensiveness and relevance. Face validity was assessed through yes/no responses, with over 80% of respondents confirming the clarity and 85% the necessity of each question, leading to the retention of all 47 questions. Data collected from the five-point responses (strongly disagree to strongly agree) for each question were used to assess the questionnaire's internal consistency. The Cronbach's alpha was 0.94, indicating a high internal consistency. CONCLUSION: This self-assessment questionnaire is evidence-based and on preliminary testing is deemed valid, comprehensible and reliable. Future work should assess the range of survey responses in a large sample of respondents from different hospitals.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.071 | 0.114 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.007 | 0.004 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.002 | 0.003 |
| Open science | 0.001 | 0.002 |
| Research integrity | 0.001 | 0.002 |
| Insufficient payload (model declined to judge) | 0.005 | 0.003 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".