Too many rating scales: Not enough validation

Ghitza and colleagues 1 set out to generate debate on what screening and initial assessment tools should be used in primary care to detect substance misuse disorders. There is a long tradition in the addictions field of bringing together expert groups to find a consensus on some aspect of data collection and, invariably, the conclusion is that there should be a variety of measures available to suit different needs 2. An exception was the Diagnostic and Statistical Manual (DSM), but now controversies about the draft DSM-V revision, including new behavioural addictions and differences on the meaning of addiction, have caused a rift between Europeans and Americans 3. Ghitza and colleagues 1 present work tailored to the demands of Medicare and Medicaid services in the USA with barely a hint at its relevance to other countries. Nonetheless, the project provokes reflection regarding the suitability of rating scales. One reason for the inability of expert groups to reach meaningful conclusions is the process itself. Experts and stakeholders are, by definition, selected because they are distinguished in their field, and have opinions and experience to bring to the table. The problem is that opinions are often very strongly held to the exclusion of equally strong science 4. The method described by Ghitza and colleagues 1 for building their consensus is an example—it is understandable, but strangely unscientific. A systematic review 5, 6 would have strengthened the starting position but herein lies a second reason—the lack of validation studies—for reaching, at best, only tentative or preliminary conclusions and, at worst, compromised or misleading ones. Take as examples the single screening question for illegal drugs 7 which performed as well as the longer Drug Abuse Screening Test (DAST) 8 and the single alcohol question 9—tested in the same population—which was better on sensitivity but less good on specificity compared to the Alcohol Use Disorders Identification Test (AUDIT)-C. These are two excellent studies of two single question screeners; however, they are also the only studies and this is the nub of the argument suggesting that the evidence bar needs to be raised before validity is accepted. The sample completing interviews was 286 from 1781 people approached from which the alcohol question delivered useful categories of drinker, but the drugs question simply identified current use or use disorder. There would be no question of approving a pharmacotherapy on the basis of such flimsy data so why allow it for important measures that are expected to influence policy? Happily, there is no need for an expert group to consider what criteria constitute a truly robust validation. The psychometric properties required of a screening or assessment instrument are well known 10. Less attention has been given to more general quality markers. Important among these are the number and diversity of independent reports, the readability of scales, the availability (costs and media formats), service user acceptability and the utility as part of a package of measures 11. In a systematic review 5 the DAST was identified as one of the most widely studied measures—seven articles were found evaluating the DAST as a screening tool, of which five were with psychiatric populations and studies were undertaken in the USA, Canada and India. Both the 28- and the 10-item versions were found to have good psychometrics. The review concluded, notwithstanding the evidence in support of the DAST, that there is insufficient comparative evidence to choose between several available measures and a decision to use the Alcohol, Smoking and Substance Abuse Screening Test (ASSIST) 12, for example, could well be equally justified. Wherever possible it makes sense to use assessment measures that will also be outcome measures. The UK government has encouraged the use of patient-reported outcome measures (PROMs) 13 not least as a means of determining levels of payment to treatment providers. Essentially, PROMs can be divided into those that are generic and those that have some degree of specificity, either related to a treatment specialty or particular disorder 14. Typically, PROMs are self-completion questionnaires—nothing new to the addictions field, but novel in some areas of healthcare. The need is for fewer instruments and very much more testing of the best ones. Notably, there is a paucity of data on service user views, for example how important a particular domain is, how accurately questions are understood and how comfortable is it to answer the questions. Of course, validation is an ongoing process and tools are needed now even though they may be imperfect. But, if we had progressed development of the tape measure to be only roughly accurate and only in some situations would we call it a tape measure? None.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.