Bibliographic record
Abstract
Sir: The strength of systematic reviews relies inherently on the methodology of the included studies, which inevitably to attain a high level of evidence must be reliable and transparent. We read with great interest a study conducted by Momeni et al.,1 in which the authors evaluated the methodologic quality of systematic reviews in the hand surgery literature using the validated critical appraisal tool AMSTAR.2 The authors concluded that the median AMSTAR overall score of systematic reviews in the hand surgery literature was 7 of 11, which corresponds to a fair to good score. However, we noted that the study has two major underlying issues. First, the authors stated that the majority of the evaluated systematic reviews provided only a list of included studies, but none provided a list of excluded studies. The fifth criterion in the AMSTAR tool states that fulfillment of this criterion requires “a list of included and excluded studies should be provided.”2 However, the authors stated that providing solely a list of included studies was sufficient to satisfy this AMSTAR criterion. Modifying the original criterion of AMSTAR can potentially negatively influence the validity of the AMSTAR tool and thus lead to misleading results. Second, the last criterion of the AMSTAR tool, which consists of evaluating conflict of interest, was reported in 81 percent of the included systematic reviews. This criterion states that “potential sources of support should be clearly acknowledged in both the systematic review and the included studies.”2 However, the authors consider acknowledgment of source of support in systematic reviews only and not the included studies. Popovich et al. found that more than half of Cochrane reviews did not assess conflicts of interest in their included studies, which resulted in a downshift of final score.3 Although we acknowledge this is a difficult criterion to achieve, we feel that this criterion should not be counted as “yes” if not fully met, as it could result in a misleadingly high-quality score. We conducted a pilot systematic search, which involved a MEDLINE search of all hand surgery systematic reviews from 2003 to 2013 published in Plastic and Reconstructive Surgery. The search identified a total of 90 articles. After screening titles and abstracts, 76 articles were excluded. Eventually, 14 hand surgery systematic reviews were identified. We applied the same quality assessment tool, AMSTAR, on these studies, focusing on the two aforementioned criteria discussed previously. We found that only 21.4 percent (versus 88.1 percent) of studies provided a list of included and excluded studies and none of the studies (versus 81 percent) provided a clear acknowledgment of potential source of support in the systematic review and the included studies. Researchers and clinicians have the responsibility of providing the highest possible level of evidence, given the significant implications on clinical and policy decision-making. When we assess the literature, the aim is to ensure that the best quality and highest level of evidence is being used. We should avoid overstating any results, as this could affect the progress of the improvement of quality in our surgical specialty by giving a false encouraging image. We find that this unjustified modification of the two AMSTAR criteria has led to inflated results that may have influenced the final conclusion of the study in a misleading manner. Although we feel that these modifications will not help in the improvement of the quality of systematic reviews in the hand surgery literature, the authors may have a reasonable justification for modifying these two AMSTAR criteria. DISCLOSURE The authors have no financial interest to declare in relation to the content of this communication. This letter did not require any funding. Osama A. Samargandi, M.D., M.H.Sc. School of Population and Public Health Faculty of Medicine University of British Columbia Vancouver, British Columbia, Canada Haroon Hasan, B.Sc., M.P.H. Department of Radiation Oncology British Columbia Cancer Agency Vancouver, British Columbia, Canada and Pediatric Oncology Group of Ontario Toronto, Ontario, Canada
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.409 | 0.703 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.028 | 0.009 |
| Bibliometrics | 0.002 | 0.002 |
| Science and technology studies | 0.000 | 0.001 |
| Scholarly communication | 0.001 | 0.000 |
| Open science | 0.002 | 0.000 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.002 | 0.002 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; both teacher heads agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".