Serious Adverse Events: A Replicability and Validation Study of Naranjo Causality Assessment Tool in a Canadian Clinical Setting
Bibliographic record
Abstract
ABSTRACT Rationale Patient safety has become a major concern in healthcare today as 5%–10% of patients experience serious adverse events (SAE) during their hospital stay. The causal assessment of SAE is the responsibility of healthcare professionals (HCP), who use their judgment or a standardize tool. Whether those two methods are replicable to provide similar results remains unclear. Objective Our aim was to evaluate if causality assessment performed by HCP is replicable when systematically assessed with the Naranjo tool and to validate its performance in Canadian clinical context. Methods We performed pilot retrospective cohort study which included patients with SAE admitted to a Quebec hospital in 2021. Twelve SAE were randomly selected, and two reviewers independently assessed their causality using Naranjo tool. Inter‐rater reliability among two reviewers and between HCP was evaluated. Along with criterion validity, sensitivity and specificity were calculated for validation study. Results Weighted kappa was 0.92 (good inter‐rater reliability) where kappa was 0.84 (good agreement between reviewers). No causality assessment by HCP was documented leading to impossibility in computing replicability. The Naranjo tool showed positive monotonic correlation with expert opinion resulting in rs = 0.208 (p < 0.001). Classification of Naranjo scores to binary variables resulted in sensitivity of 1.00 and specificity of 0.31. Conclusion Our study suggested that Naranjo tool is reliable and valid to be used in a clinical setting and was able to classify all drug products involved in the occurrence of SAE. Larger scale studies need to be conducted in real‐time clinical settings to investigate its performance and utility.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.206 | 0.316 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.002 | 0.003 |
| Bibliometrics | 0.004 | 0.004 |
| Science and technology studies | 0.004 | 0.004 |
| Scholarly communication | 0.003 | 0.002 |
| Open science | 0.003 | 0.002 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.002 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; the direct Gemma label and the distilled Codex classifier agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".