Assessing the evidence on Artificial Intelligence: Health technology assessments and real-world needs for decision-making in health care
Bibliographic record
Abstract
INTRODUCTION Artificial Intelligence (AI) is rapidly entering the field of medicine. With many AI tools available, an increased interest in understanding and identifying effective AI applications for different aspects of healthcare delivery is reported by decision-makers in Canada. THE PRESENTATION This presentation will provide an overview of recent evidence reviews by the Canadian Agency of Drugs and Technologies in Health (CADTH) on AI technologies for two different fields in healthcare: lung cancer diagnosis (1) and mental health (in collaboration with the Mental Health Commission of Canada) (2). It will highlight the process for evaluating the evidence on the clinical utility of AI applications, the diagnostic accuracy, cost-effectiveness and guidelines for clinical use. The presentation will also cover high-level findings of a pan-Canadian survey that collected information about the extent and type of use of AI in imaging departments across Canada (3). OBJECTIVES The objective of this presentation is to highlight current evidence and real-world use of various applications of AI in healthcare. In addition to the main findings of the evidence reviews, the presentation will aim to invite a discussion on implications of the findings for policy and clinical decisions and the utility and value of such reviews for addressing the needs of healthcare decision-makers in the context of the rapidly evolving field of AI. REFERENCES Artificial intelligence for classification of lung nodules: clinical utility, diagnostic accuracy, cost-effectiveness and guidelines. (CADTH rapid response report: summary with critical appraisal). Ottawa: CADTH; 2020 Jan. Artificial Intelligence and Machine Learning in Mental Health Services: A Literature Review. Ottawa: CADTH; 2020 Aug. (CADTH Rapid Response Report: Summary With Critical Appraisal). Joint publication with the Mental Health Commission of Canada. The Canadian Medical Inventory 2019-2020. Ottawa: CADTH; 2021 Jan. (CADTH health technology review).
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.198 | 0.552 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.004 | 0.004 |
| Bibliometrics | 0.021 | 0.019 |
| Science and technology studies | 0.002 | 0.009 |
| Scholarly communication | 0.018 | 0.009 |
| Open science | 0.004 | 0.005 |
| Research integrity | 0.007 | 0.007 |
| Insufficient payload (model declined to judge) | 0.016 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; the direct Gemma label and the distilled Codex classifier agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".