Expert Panels: Can They Be Trusted?

Van Meer et al.1van Meer B.L. Meuffels D.E. Vissers M.M. et al.Knee Injury and Osteoarthritis Outcome Score or International Knee Documentation Committee Subjective Knee Form: Which questionnaire is most useful to monitor patients with an anterior cruciate ligament rupture in the short term?.Arthroscopy. 2013; 29: 701-715Abstract Full Text Full Text PDF PubMed Scopus (67) Google Scholar recently compared the Knee Injury and Osteoarthritis Outcome Score (KOOS)2Roos E.M. Roos H.P. Lohmander L.S. et al.Knee Injury and Osteoarthritis Outcome Score (KOOS)—Development of a self-administered outcome measure.J Orthop Sports Phys Ther. 1998; 28: 88-96PubMed Google Scholar with the International Knee Documentation Committee (IKDC) Subjective Knee Form3Irrgang J.J. Anderson A.F. Boland A.L. et al.Development and validation of the International Knee Documentation Committee Subjective Knee Form.Am J Sports Med. 2001; 29: 600-613PubMed Google Scholar to investigate which of the instruments is most useful after anterior cruciate ligament reconstruction (ACLR). Following the COSMIN (COnsensus-based Standards for the selection of health status Measurement INstruments) checklist, an expert panel reached consensus on a priori hypotheses regarding responsiveness and construct validity. This letter raises concerns about the use of expert panels in general and gives examples of bias introduced by the expert panel in this particular study. van Meer et al. state: “Without specific hypotheses there is a risk of bias, because it is tempting to formulate explanations for the low and high correlation coefficients retrospectively instead of concluding that the questionnaire may not be valid. Conversely, the choice of magnitude of the hypotheses is arbitrary.” If the expert panel had considered available literature when establishing a priori hypotheses, their results would have been different. Using an expert panel is thus not necessarily associated with less bias compared with relying on available literature. A minimum requirement for an expert panel must be awareness of existing literature when formulating a priori hypotheses. Especially evident for KOOS, correlations hypothesized are not in line with existing literature. In contrast, the majority of the correlations actually found for both KOOS and IKDC are in line with existing literature. One example relates to effect size (ES). For the IKDC, van Meer et al. hypothesized an ES >0.8, found an ES of 1.36, and the hypothesis was confirmed, well in line with existing literature (ES, 0.76 to 2.11).4Collins N.J. Misra D. Felson D.T. et al.Measures of knee function: International Knee Documentation Committee (IKDC) Subjective Knee Evaluation Form, Knee Injury and Osteoarthritis Outcome Score (KOOS), Knee Injury and Osteoarthritis Outcome Score Physical Function Short Form (KOOS-PS), Knee Outcome Survey Activities of Daily Living Scale (KOS-ADL), Lysholm Knee Scoring Scale, Oxford Knee Score (OKS), Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC), Activity Rating Scale (ARS), and Tegner Activity Score (TAS).Arthritis Care Res (Hoboken). 2011; 63: S208-S228Crossref PubMed Scopus (693) Google Scholar For KOOS, however, it seems that the predefined hypotheses were arbitrary. For the subscales Pain, activities of daily living (ADL), and quality of life, ESs less than 0.4 were hypothesized. The ESs found (0.58 to 1.51) are, however, in line with existing literature (ACLR, 0.84 to 1.65).2Roos E.M. Roos H.P. Lohmander L.S. et al.Knee Injury and Osteoarthritis Outcome Score (KOOS)—Development of a self-administered outcome measure.J Orthop Sports Phys Ther. 1998; 28: 88-96PubMed Google Scholar, 4Collins N.J. Misra D. Felson D.T. et al.Measures of knee function: International Knee Documentation Committee (IKDC) Subjective Knee Evaluation Form, Knee Injury and Osteoarthritis Outcome Score (KOOS), Knee Injury and Osteoarthritis Outcome Score Physical Function Short Form (KOOS-PS), Knee Outcome Survey Activities of Daily Living Scale (KOS-ADL), Lysholm Knee Scoring Scale, Oxford Knee Score (OKS), Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC), Activity Rating Scale (ARS), and Tegner Activity Score (TAS).Arthritis Care Res (Hoboken). 2011; 63: S208-S228Crossref PubMed Scopus (693) Google Scholar Expert clinicians and patients evaluated content validity. This approach may introduce further bias because the perspectives of clinicians and patients differ and current standards consider the patient to be the expert. The importance of focusing on the patient's perspective is underscored by the findings in the van Meer study. Although the authors concluded that the KOOS subscales Pain and ADL were irrelevant, they reported ES > 0.50. A moderate improvement in Pain and ADL at 1 year after ACLR is not irrelevant. Van Meer et al. state: “A disadvantage of the IKDC subjective is the use of one total score, which means it is impossible to see in which domain the patients have improved. According to our results, the IKDC subjective is more responsive to changes over time than is the KOOS.” This statement is incorrect. The ES for the different KOOS subscales ranged from 0.55 to 1.51, reflecting the difference in improvement for the 5 domains at 1 year after ACLR, whereas the ES for the IKDC subjective was 1.36. Finally, in Table 2 the correlation found between the IKDC and Lysholm score (0.47) is wrongly interpreted as being higher than the hypothesized correlation of 0.6. Overall, 75% of the hypothesized and the found correlations are not given in the tables and can thus neither be compared with the previous literature nor checked for correctness. Knee Injury and Osteoarthritis Outcome Score or International Knee Documentation Committee Subjective Knee Form: Which Questionnaire Is Most Useful to Monitor Patients With an Anterior Cruciate Ligament Rupture in the Short Term?ArthroscopyVol. 29Issue 4PreviewTo evaluate which questionnaire, the Knee Injury and Osteoarthritis Outcome Score (KOOS) or the International Knee Documentation Committee Subjective Knee Form (IKDC subjective), is most useful to evaluate patients with recent anterior cruciate ligament (ACL) ruptures or those within 1 year of an ACL reconstruction. Full-Text PDF Author's ReplyArthroscopyVol. 29Issue 7PreviewWe thank Roos, Juhl, and Lohmander for their interest in our article and their remarks. They raise concern about the use of an expert panel for determining a priori hypotheses for the assessment of construct validity and responsiveness. They argue that the expert panel should have considered existing literature when formulating a priori hypotheses. In contrast, we do not think that hypotheses should be based on the results of previous validation studies. Classic psychometric works state that hypotheses should be based on theoretical considerations about expected relations among constructs. Full-Text PDF

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.