True North: Building Imaginary Worlds with the Revised Canadian (CADTH) Guidelines for Health Technology Assessment
Bibliographic record
Abstract
In March 2017 the Canadian Agency for Drugs and Technologies in Health (CADTH) released the 4th edition of their Guidelines for the Economic Evaluation of Health Technologies: Canada. These guidelines, which were first published and revised for a 3rd edition in 2006 are intended to help decision makers, health systems leaders and policy makers make well-informed decisions. They are designed, apparently, to support best practice in conducting health technology assessments in Canada. The purpose of this commentary is to consider whether or not the evidence standards proposed and the consequent modeled claims for economic effectiveness meet the standards of normal science: are the CADTH standards capable of generating claims for competing products that are credible, evaluable and replicable? The review argues that the standards proposed by CADTH do not meet the standards expected in normal science. Technical sophistication in building reference case imaginary worlds is not a substitute for claims that are experimentally evaluable or capable of assessment through systematic observation. There is no way of judging whether imaginary claims are right or even if they are wrong. CADTH is not alone in setting standards that fail to meet the standards of normal science. Recent commentaries on formulary submission guidelines in a number of other countries, to include Ireland, the Netherlands, France, Australia, the UK and New Zealand conclude that they are subject to the same criticism. If the CADTH guidelines were never intended to support feedback to health system decision makers, then this should be made clear. If not, then consideration should be given to withdrawing the guidelines to ensure they conform to these standards. Hopefully, future versions of the CADTH guidelines will address this issue and focus on a rigorous research program of claims assessment and feedback and not the building of imaginary worlds.
 
 Type: Commentary
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.010 | 0.003 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.000 |
| Bibliometrics | 0.001 | 0.001 |
| Science and technology studies | 0.001 | 0.000 |
| Scholarly communication | 0.000 | 0.001 |
| Open science | 0.001 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".