OPEN SCIENCE POLICIES AND CORE OPEN SCIENCE PRACTICES IN MAJOR MEDICAL JOURNALS: DEVELOPMENT OF AN AUTOMATIZED TOOL BASED ON A LLM

Open science policies and core open science practices in major medical journals: development of an automatized tool based on a LLM GUILLAUME FREYERMUTH / CONSTANT VINATIER / GAUTHIER LE-BARTZ-LYAN / SARAH BUET / PERRINE LUNEL / NICOLAS ROBILLARD / GWENAEL DUMONT / FLORIAN NAUDET / MATHIEU ACHER BACKGROUND Monitoring the use and application of Open Science practices is key to fostering Open Science and to understanding barriers and facilitators of open science practices. Monitoring also aids in measuring the impact of open science policies and can serve as a possible intervention to improve transparency and reproducibility. For instance, the Hong Kong principles support the assessment of researchers on the use of Open Science practices. Open Science Monitoring is a key priority for organizations such as the UNESCO [and the G7. The scientific community is also beginning to agree on key indicators to measure. For instance, in biomedicine, a core set of 19 open science practices has been developed to monitor various open science practices such as preregistration, open data, open access, open code, conflict of interest statements etc. In recent years, biomedical journals have implemented some policies that promote open science practices. Most notably, the influential International Committee of Medical Journal Editors (ICMJE) made prospective registration a requirement to publish clinical trial results in 2005. In 2018, it required a specific data-sharing statement to be included in each newly submitted paper (and prespecified in study registration). The impact of these policies have been studied previously showing an major influence on trial registration practices, and mixed results regarding actual data sharing. Automatized options are being developed to track change within the biomedical literature. It is therefore useful to assess the impact of the various open science policies adopted by biomedical journals using those tools. OBJECTIVES This aim of this study, targeting major medical journals, is three-fold: 1. to describe open science policies in these journals, 2. to develop and validate automatic tools that can be used to monitor key transparency practices 3. to describe retrospectively changes on open science and transparency practices for articles published in those journals. METHODS Selection criteria We will include in our study a sample of 10 major medical journals. Population: We will include research studies published in 2020 or 2021 in journals of interest: - Annals of internal medicine - BioMed Central Medicine - The British Medical Journal - Canadian Medical Association Journal - Journal of the American Medical Association - JAMA Network Open - Lancet - Nature Medicine - Public Library Of Science Study design: All types of research articles will be included with a proportional number of 1/RCTs, 2/ meta-analyses and 3/ other research articles. Data sources: All articles and supplementary materials will be extracted from Pubmed and from the journal website. If an article isn’t open access, academic access will be used to get it. Automatized tool Here we want to develop a tool using machine learning and to explore the feasibility of a tool based on a large language model (LLM) such as Mixtral or llama to extract the presence and absence of open science practices inside. Ideally this tool should be automatized, open source (or failing that, open weights). and used locally to enforce reproducibility. It should be able to identify journal articles (either in XML or pdf format) and all supplementary material (in pdf format) and to extract core Open Science practices. The task of extracting content out of a scholar article with an LLM involves optional innovations to ensure accuracy, robustness, and efficiency, such as embedding of raw data into vector stores, refining the method of prompting the LLM, and employing techniques like retrieval augmented generation (RAG) or fine-tuning. Creation of the validation database: Data collection occurred from February 25th to March 18th 2024, following a standardized approach. For each journal, a stratification about study type was realized to select 10 RCT, 10 meta research and 10 other research articles per journal. To perform article selection, 1/ the algorithm of research from PubMed was used as described in table 1. Then, in each journal/category, we randomly selected 10 articles. As it, the database is composed of 30 articles from each journal with an equal representation of the different type of study designs for a total of 300 articles. In case the required number of articles of a given type wasn’t available in a given journal, the total number of articles was completed by articles with the same study design published in other journals (that were each time from a different journal). This was done by a first group of master 1 students (SB, PL, NR). Search strategy and article selection process All articles were classified by the publication date, the latest will get the number 1 and the oldest the max. After assigning these numbers, for each journal and type of study, each was given a random number and eligibility of articles will be assessed – for each journal and type of study- using this random number and included if those are in line with our selection criteria (data and code available on the Open Science Framework). As soon as the target number of included articles was reached, the same selection process was applied for the next journal and type of study. Table 1: research algorithm for each study type: Study type Research algorithm RCT (2020:2021[Date - Publication] AND "hasabstract" AND randomizedcontrolledtrial[Filter] AND ("Annals of Internal Medicine"[Journal] OR "BMC Medicine"[Journal] OR "BMJ"[Journal] OR "CMAJ"[Journal] OR "JAMA"[Journal] OR "JAMA Netw Open"[Journal] OR "Lancet"[Journal] OR "Nat Med"[Journal] OR "N Engl J Med"[Journal] OR "PLOS Med"[Journal])) Meta research (2020:2021[Date - Publication] AND "hasabstract" AND meta-analysis[Filter] AND ("Annals of Internal Medicine"[Journal] OR "BMC Medicine"[Journal] OR "BMJ"[Journal] OR "CMAJ"[Journal] OR "JAMA"[Journal] OR "JAMA Netw Open"[Journal] OR "Lancet"[Journal] OR "Nat Med"[Journal] OR "N Engl J Med"[Journal] OR "PLOS Med"[Journal])) Other research article ((2020:2021[Date - Publication]) AND "hasabstract" NOT ("comment"[Publication Type]) NOT ("letter"[Publication Type]) NOT ("editorial"[Publication Type]) NOT ("published erratum"[Publication Type]) NOT ("news"[Publication Type]) NOT ("introductory journal article"[Publication Type]) NOT ("biography"[Publication Type]) NOT ("portrait"[Publication Type]) NOT ("congress"[Publication Type]) NOT ("interview"[Publication Type]) NOT ("retraction of publication"[Publication Type]) NOT ("personal narrative"[Publication Type]) NOT ("retracted publication"[Publication Type]) NOT ("patient education handout"[Publication Type]) NOT ("lecture"[Publication Type]) NOT ("autobiography"[Publication Type]) NOT ("clinical conference"[Publication Type]) NOT ("classical article"[Publication Type]) NOT ("address"[Publication Type]) NOT ("legal case"[Publication Type]) NOT ("expression of concern"[Publication Type]) NOT ("festschrift"[Publication Type]) NOT ("overall"[Publication Type]) NOT ("bibliography"[Publication Type]) NOT ("corrected and republished article"[Publication Type]) NOT ("interactive tutorial"[Publication Type]) NOT ("duplicate publication"[Publication Type]) NOT ("directory"[Publication Type]) NOT ("newspaper article"[Publication Type]) NOT ("periodical index"[Publication Type]) NOT ("dictionary"[Publication Type]) NOT ("meta analysis"[Publication Type]) NOT ("randomized controlled trial"[Publication Type])) AND ("Annals of Internal Medicine"[Journal] OR "BMC Medicine"[Journal] OR "BMJ"[Journal] OR "CMAJ"[Journal] OR "JAMA"[Journal] OR "JAMA Netw Open"[Journal] OR "Lancet"[Journal] OR "Nat Med"[Journal] OR "N Engl J Med"[Journal] OR "PLOS Med"[Journal]) Each research algorithm was validated according to the PRESS guideline (1) by an information specialist (XCH). Description of tools’ performance Development will use the validation database of 300 papers selected from articles published in the journals of interest during the year 2020-2021 (30 per journal). Statistical analysis: Tool’s performance will be determined by the computation of metrics: sensitivity, specificity, PPV, NPC and the F1 score and the plotting of the receiver operating characteristic curve (ROC). As highlighted in the literature (2), any error may also come from human. For each disagreement, a manual verification of the result by an expert (CV or FN) will be done to determine if the LLM is truly false or not. Outcomes to be extracted both for training and validation - Type of study (Randomized Control Trial, meta-analyses, network meta-analysis, Observational study) - Pre-registration (Yes/No) - Data sharing (fully open access, on request, not available) - Open Access Publishing (Yes/No) - Code Sharing (Yes/No) - Publication of results within 1 year of study completion (RCT) (Yes/No) - Materials sharing (STAP, Protocol) (Yes/No) - Utilization of reporting guideline (Yes/No) + which reporting guideline is use - Reporting the existence of a preprint (Yes/No) - Authors contribution (Yes/No) - COI statement (Yes/No) - Reporting ORCID identifiers (Yes/No) - Funding statement (Yes/No) Implications We aim to develop automated tools to monitor open science practices at major generalist biomedical journals. This work will inform the development of a broader open science observatory of biomedical journals. The leading biomedical journals in this study will hence provide important information about these journals that are considered as trusted sources by the medical community. Last, by tracking the open science practices, we will be able to describe the impact of various ICJME policy changes on selected open science practices (

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.