Efforts to Limit Publication Bias and Improve Quality in the Journal

Interventional pulmonology (IP) has continued to grow as a discipline, with significant advances in technique, knowledge, and scholarship over the last 2 decades. Central to this progress are the 3 pillars of IP: innovation, education, and academic investigation.1 The role of the Journal of Bronchology and Interventional Pulmonology in this context is to facilitate the exchange and dissemination of reliable, high-quality information to advance the knowledge frontier of IP. This, in turn, will hopefully improve patient care and outcomes. The information conveyed in the Journal is primarily developed by researchers and practitioners in the IP field. In a sense, the Journal serves as a market of ideas for the discipline. Within this market, different members of the IP community advance and present their ideas, hypotheses, observations, and results. Because there are too many ideas being advanced for readers/evidence-users to evaluate all of them, the Journal provides value by critically evaluating all submissions, prioritizing, and disseminating the most meritorious reports. The ultimate decision-maker in terms of practice is not the editorial board, but rather the practitioner/evidence user who has access to other information sources as well. However, journals, their editorial boards, and the editorial process influence what ideas receive more attention as not all manuscripts can or should be published. Therefore, a mechanism for selecting the most meritorious manuscripts is required. The scientific peer review process forms the backbone of this selection process. The origins of the scientific peer review process date back to Henry Oldenburg, one of the original fellows of the Royal Society founded in 1660.2 Oldenburg, a theologian by training as well as a natural philosopher and diplomat, was the founding editor of Philosophical Transactions, established in 1665, as a private venture of the Royal Society. The title changed to the more commonly recognized Philosophical Transactions of the Royal Society in 1776. This was the first journal exclusively devoted to science in the world. Oldenburg began the practice of sending manuscripts to experts who could judge their quality before publication and this evolved into the peer review process. In the United States, the scientific peer review process dates back to the early 1900s, with JAMA, Science, and American Practitioner being the first journals to implement it.3 In its current form, the goals of the peer review process include evaluating the quality of the information provided, the methodologic rigor of the investigation, and improving the quality of manuscripts by identifying flaws and gaps. The modern scientific publication process utilizes peer review as a key component for defining which manuscripts merit dissemination. However, the publication process does have important limitations, and it is important that evidence-users, investigators, and editors know these limitations and work to improve the system. One of the main concerns related to the publication process is the potential for bias to enter into the system.3 In this context, we define bias as a violation of impartiality in the evaluation of a submission that results in a distortion of the results from the perspective of evidence-users. Rejection constitutes a change in results from the evidence-users perspective because the manuscript cannot be read. However, more subtle distortions can also occur. For purposes of this discussion, we will call this impartiality bias. Bias in this context may be hard to detect for the evidence-user because the bias results in some manuscripts not being published, making them unavailable to evidence-users. Impartiality bias is subtly different from other definitions of bias, which imply a unidirectional systematic distortion of results from the truth. Impartiality bias is often confused or combined with issues that lower the quality of the review process, while not necessarily meeting the definition. For example, a reviewer who lacks content knowledge in all areas does not meet this definition of bias, as they impartially apply their lack of knowledge uniformly to all reviews, but it does negatively impact the quality of the review and ultimately the advancement of science if decisions are based on this poor quality review. If the editor randomly assigns this reviewer, impartiality bias is not created even if the reviewer’s recommendations are followed. However, there is potential that the lack of knowledgeable reviewers may create unidirectional systematic distortions in the body of evidence while still being impartial. These dual problems of impartiality bias and poor quality reviews can enter into the publication process at many points. One conceptually useful framework is to classify these issues based on the time they occur in the process—either before, during, or after peer review (Fig. 1).3FIGURE 1: Publication review process and potential mechanisms for introducing bias.Before peer review, editors and associate editors review manuscripts and decide which manuscripts are worthy of going on to further peer review. If editors reject worthwhile manuscripts before peer review for reasons other than flaws in methodologic rigor, importance, or relevance, then bias may enter the system. Editors can also create impartiality bias by intentionally choosing reviewers known to be either more or less favorably disposed on a given topic or by choosing reviewers known to be either more or less lenient in general. During peer review, a variety of problems can occur, some of which relate to bias, whereas others relate to the quality of reviews (Table 1). Bias may also occur after peer review. Examples include delays in a publication that are conditional on the results of the study or editorial positions that are intentionally published concurrently with original research, chosen to specifically support or reject a given position.TABLE 1: Potential Problems in Peer ReviewDifferent systems of peer review can deal with some but not all of these issues. Systems of peer review include single-blind, double-blind, triple-blind, quadruple-blind, and open-review (Table 2). These systems vary in terms of whether authors, reviewers, and editors know the identity of each other. No one system has proven clearly superior. Studies comparing different forms of peer review are very limited and most have significant technical limitations, so caution is warranted when drawing inferences from this data.5 A meta-analysis of randomized controlled studies comparing the impact of different interventions in the peer review process failed to demonstrate a significant difference between single and double-blind peer review systems in terms of review quality and rejection probability.4 Open peer review with unblinded authors improved the quality of reviews (3 studies) and decreased the probability of rejection (n=4).6–13 Two randomized controlled trials were identified that evaluated the impact of adding a statistical peer reviewer, suggesting that it improved the quality of reviews. Use of reviewer checklists failed to demonstrate an impact on review quality.TABLE 2: Systems of Peer ReviewGiven this context and the limited evidence, the Journal of Bronchology and Interventional Pulmonology has implemented several changes over the last year to improve the Journal’s publication process. The most significant of these is the introduction of a double-blind peer review system. All new manuscripts beginning in April 2019 have used a double-blinded peer review process. Double-blind peer review systems address bias introduced by reviewers, whose knowledge of manuscripts’ authors may influence their decision (eg, academic rank, stature, prior publication, personal relationships). Double-blind peer review is not perfect, as authors may accidentally give away their identity (eg, referencing their institution in the methods or their prior publications), but on balance, it appears preferable to single-blinded peer review systems. Although open peer review has been advocated by some, the potential benefits must be weighed carefully against the harms. Benefits such as having “skin in the game” and evidence suggesting improved review quality must be weighed against the potential harms. The potential for worsening retaliatory events and animosity between investigators both within the journal but also outside of the journal (eg, when authors are given the chance to review their reviewers in a different journal that is single blinded) represents a significant and possibly insurmountable obstacle.14 The open peer review system is also associated with higher refusal rates from reviewers, suggesting that whether or not retaliation is real, reviewers are probably concerned about this possibility. There is a scarcity of good peer reviewers because it is uncompensated work, and an open system might exacerbate this already severe issue. In addition, the finding that open systems lead to lower rejection rates is consistent with the hypothesis that open systems might encourage reviewers to lower their standards due to concerns over retaliation or relationships. Open peer review would also probably favor senior investigators from larger institutions over those more junior in rank from smaller institutions as the perceived potential for meaningful retaliation would differ substantively between groups. In addition, over time, systems tend to produce more of the things they reward. Among investigators of similar stature from similar institutions, the open system would favor those individuals known to be more likely to retaliate, in essence rewarding certain undesirable behavioral patterns. Such subtle cultural effects might be difficult to detect initially, but over the long term might have large negative impacts on the culture of science, scholarship, and collegiality within the IP community. On balance, while the benefits of open peer review are nontrivial, it is the opinion of the editors that the magnitude of the potential for harm outweighs the benefits given the currently available evidence. Triple-blind and quadruple-blind systems, while potentially better, create operational complexities and increase cost, as it is difficult if not impossible for administrative staff to determine if the reviewers selected are definitely not in some way related to the authors of a given manuscript. Although such systems can work, they require additional volunteer labor from physicians, making such solutions less tenable and they may potentially lead to delays in the review process. The Journal has also introduced new biostatistical editorial board members. These biostatisticians serve as resources for, particularly complex manuscripts. Although there are too few resources to have a biostatistician for every manuscript, this is especially useful for select manuscripts and improves review quality as suggested in the literature.6 Studies in IP increasingly use complex methods of analysis, such as meta-analysis, propensity matching, time to event analysis, noninferiority trials, cost-effectiveness, and quality-adjusted survival.15–19 Adding this dimension hopefully will improve the value delivered to readers. To address post peer review issues, the Journal has moved to a system where all accepted manuscripts are published online ahead of print as soon as possible. This rewards author and addresses the potential for post peer review delay in publication due to bias. To address issues of review quality, each original article is first reviewed by the editor, then 1 associate editor, then 1 editorial board member who is as a blinded reviewer along with ≥1 external blinded peer reviewers. The presence of noneditorial board members insures a diversity of perspective, whereas the inclusion of at least 1 editorial board member help to insure at least 1 quality review. The editorial board member selection process has been revised with this goal in mind. Board member invitation is now based on prior publications of original research, formal training and/or experience in clinical research methodology, and submission of high-quality timely reviews to the Journal (all reviews are now scored for quality). In summary, the Journal continues to evolve in parallel with the IP field. Introduction of double-blind peer review and the other changes described above will hopefully incrementally improve the quality of the publication and peer review process to minimize bias and maximize quality. This, in turn, will hopefully facilitate the advancement of the knowledge frontier in IP which will lead to improved patient outcomes in the future.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.