Evaluating Approaches to Quality Assessment in Library and Information Science LIS Systematic Reviews: A Methodology Review
Notice bibliographique
Résumé
Objective – Systematic reviews are becoming increasingly popular within the Library and Information Science (LIS) domain. This paper has three aims: to review approaches to quality assessment in published LIS systematic reviews in order to assess whether and how LIS reviewers report on quality assessment a priori in systematic reviews, to model the different quality assessment aids used by LIS reviewers, and to explore if and how LIS reviewers report on and incorporate the quality of included studies into the systematic review analysis and conclusions. Methods – The authors undertook a methodological study of published LIS systematic reviews using a known cohort of published systematic reviews of LIS-related research. Studies were included if they were reported as a “systematic review” in the title, abstract, or methods section. Meta-analyses that did not incorporate a systematic review and studies in which the systematic review was not a main objective were excluded. Two reviewers independently assessed the studies. Data were extracted on the type of synthesis, whether quality assessment was planned and undertaken, the number of reviewers involved in assessing quality, the types of tools or criteria used to assess the quality of the included studies, how quality assessment was assessed and reported in the systematic review, and whether the quality of the included studies was considered in the analysis and conclusions of the review. In order to determine the quality of the reporting and incorporation of quality assessment in LIS systematic reviews, each study was assessed against criteria relating to quality assessment in the PRISMA reporting guidelines for systematic reviews and meta-analyses (Moher, Liberati, Tetzlaff, Altman, & The PRISMA Group, 2009) and the AMSTAR tool (Shea et al., 2007). Results – Forty studies met the inclusion criteria. The results demonstrate great variation on the breadth, depth, and transparency of the quality assessment process in LIS systematic reviews. Nearly one third of the LIS systematic reviews included in this study did not report on quality assessment in the methods, and less than one quarter adequately incorporated quality assessment in the analysis, conclusions, and recommendations. Only nine of the 26 systematic reviews that undertook some form of quality assessment incorporated considerations of how the quality of the included studies impacted on the validity of the review findings in the analysis, conclusion, and recommendations. The large number of different quality assessment tools identified reflects not only the disparate nature of the LIS evidence base (Brettle, 2009) but also a lack of consensus around criteria on which to assess the quality of LIS research. Conclusion – Greater clarity, definition, and understanding of the methodology and concept of “quality” in the systematic review process are required not only by LIS reviewers but also by editors of journals in accepting such studies for publication. Further research and guidance is needed on identifying the best tools and approaches to incorporate considerations of quality in LIS systematic reviews. LIS reviewers need to improve the robustness and transparency with which quality assessment is undertaken and reported in systematic reviews. Above all, LIS reviewers need to be explicit in coming to a conclusion on how the quality of the included studies may impact on their review findings.
Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.
Comment cette classification a été obtenuedéplier
Prédiction machine sur la base complète
Imitation des enseignantsNi prévalence calibrée, ni vérité terrain. Validation humaine à venir. Le volet Gemma est une étiquette directe du modèle pour chaque travail de la base, lue sur la notice réduite au titre. Le volet Codex est un classifieur appris des 10 348 étiquettes directes de Codex et calibré sur les taux pondérés de l'échantillon; les champs sans appui suffisant ne portent aucun appel Codex. Le mode candidate est l'union des deux volets; le consensus est leur intersection. Ces sorties portent le statut machine_predicted_unvalidated et ne sont pas des étiquettes humaines.
Scores du classifieur distillé par catégorie (deux têtes)
| Catégorie | Codex | Gemma |
|---|---|---|
| Métarecherche | 0,761 | 0,852 |
| Méta-épidémiologie (sens strict) | 0,005 | 0,007 |
| Méta-épidémiologie (sens large) | 0,014 | 0,020 |
| Bibliométrie | 0,039 | 0,043 |
| Études des sciences et des technologies | 0,005 | 0,009 |
| Communication savante | 0,025 | 0,017 |
| Science ouverte | 0,007 | 0,018 |
| Intégrité de la recherche | 0,007 | 0,006 |
| Charge utile insuffisante (le modèle a refusé de juger) | 0,006 | 0,001 |
Scores machine (provisoires)
Les deux têtes enseignantes du modèle étudiant, lues sur ce travail. Un score ordonne la base pour la relecture; il n'affirme jamais une catégorie, et le statut de validation accompagne chaque rangée tel quel.
Scores de référence d'un modèle non mature (critères de maturité non atteints, 7 itérations). Un score ordonne; il n'affirme jamais une catégorie.
score_only:v0-immature-baseline · tel quel depuis la passe de notation : score_only signifie que le nombre peut ordonner les travaux, et qu'aucune étiquette de catégorie n'en découleClassification
machine, non validéePrédiction automatique; l’étiquette directe de Gemma et le classifieur distillé Codex s’accordent sur ce qui est montré ici.
Le détail, modèle par modèle et score par score, se trouve en fin de page sous « Comment cette classification a été obtenue ».