Reply from Sean Williams, Richard G. Carson and Katalin Tóth

We thank Mesquida and Lakens for creating further discussion in their letter to the editor concerning our editorial, ‘Moving beyond P values in The Journal of Physiology: A primer on the value of effect sizes and confidence intervals’. We welcome the opportunity to clarify our position on these matters. Whilst the recommendation to report and interpret effect sizes in addition to P-values is indeed a century old, it remains the case that many authors submitting to The Journal of Physiology fail to do so. For example, in a recent issue of The Journal of Physiology (Volume 601, Issue 21), we could identify only one original article that reported effect sizes and confidence intervals (Trajano et al., 2023), with many reporting only the P-value. The motivation behind our editorial, therefore, was to espouse the benefits of presenting effect sizes and confidence intervals and to direct our readers to relevant resources describing how to calculate and present them. Our intention was not to dismiss the use of P-values but to encourage the reporting of effect sizes and confidence intervals alongside them. As we wrote in our editorial, ‘…it is clear that many of the limitations and misunderstandings caused by the isolated use of null hypothesis significance testing can be overcome through the complementary use of other statistics, namely effect sizes and confidence intervals’. We acknowledge that this message may not have been consistently clear throughout the editorial, and we are happy to clarify that point here. The Journal of Physiology’s statistics policy requires exact P-values to be stated, even when ‘no statistical significance’ is being reported, and there is no intention that this requirement be replaced. Complementing the reporting of results with effect sizes and confidence intervals will simply provide the reader with a more comprehensive and informative picture of the study's results, as well as facilitating future meta-analyses. When it comes to interpreting said effect sizes, the recommendation of Mesquida and Lakens that authors who submit to The Journal of Physiology specify a smallest effect size of interest (SESOI) and perform interval–hypothesis testing (e.g. equivalence tests) is one that we would support. However, it is clear that many authors submitting to The Journal of Physiology are unfamiliar with estimation approaches (i.e. effect sizes and confidence intervals) and we suspect that SESOI and equivalence testing are even less well known. Thus, the recommendation in our editorial that authors collaborate with a statistician early in the research process is one that we would highlight again here. It is unclear why Mesquida and Lakens focused specifically on exercise physiology in their letter, but it is worth remembering that The Journal of Physiology welcomes research papers in all areas of physiology and pathophysiology, from computational physiology and modelling to neuroscience and molecular and cellular physiology. Many submitted studies employ complex, factorial designs with multiple outcome measures in extremely novel domains that do not lend themselves well to an SESOI and interval–hypothesis testing approach. Thus, our statistical policy must be sufficiently flexible to accommodate fields and research questions for which specifying and interpreting effects against a SESOI may be unfeasible. It is also worth highlighting that many stakeholders may not agree with the specific choice of SESOI (and may therefore contest the interpretation and claims made of the data). But, so long as studies present effect sizes and confidence intervals, whilst also meeting The Journal of Physiology’s requirements to present all data points in a way that reveals their range and distribution and/or to provide the raw dataset with all key statistical information as supplemental files, then readers can at least draw their own conclusions on the veracity of the results. We acknowledge the limitations associated with evaluating effect sizes against field-specific thresholds, namely the lack of consideration for contextual factors and the likely inflated nature of published effect sizes (Panzarella et al., 2021). It should be noted that it is possible to adjust for publication bias when estimating effect sizes in the literature (Vevea & Hedges, 1995). Moreover, as non-significant results are substantially less likely to be written up, submitted and/or published compared to significant results (Song et al., 2010), the incorporation of estimation approaches may help alleviate the underlying issue of publication bias. Of course, wherever possible authors should consider their specific research context and apply their judgement when determining the effect size that can be considered meaningful. But, as indicated above, there are many cases in the realm of physiology for which to do so is extremely difficult. In such instances, using some standardised benchmarks may facilitate communication and comparison across studies, i.e. in the absence of a universally accepted SESOI. And so, to underline our position on this matter: many of the submissions we receive use null-hypothesis significance testing and P-values alone, which provide limited and often misleading information about the study's findings (Amrhein et al., 2019). As such, we encourage those submitting to The Journal of Physiology to also report effect sizes and confidence intervals. Our editorial and others that preceded it provide a range of excellent resources for learning about different effect size measures (Calin-Jageman & Cumming, 2019; Williams et al., 2023). We would also recommend that, where possible, authors consider the practical/clinical/physiological implications of the range of values in those intervals when interpreting their findings. Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article. No competing interests declared. Conception or design of the work; drafting the work or revising it critically for important intellectual content: S.W., R.C. and K.T. All authors have read and approved the final version of this manuscript and agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. All persons designated as authors qualify for authorship, and all those who qualify for authorship are listed. No funding was received for this work.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.