Publishing code is essential for reproducible flow cytometry bioinformatics

On behalf of the International Society for the Advancement of Cytometry, Cytometry Part A has instituted a Software Policy for manuscripts that include software for the analysis of flow cytometry data 1. These requirements include open source licensing, deposition of both code and test data, testing on standard data where applicable, and provision of all instructions necessary to install and execute code. This represents a major step forward toward reproducible flow cytometry bioinformatics. The ability to reproduce experiments is an essential part of the scientific process. For those of us working in FCM, reproducibility means providing key pieces of information, including details about the instrument used, the samples tested, and the analysis performed on the resulting data. The Minimum Information About a Flow Cytometry Experiment (MIFlowCyt) provides a set of guidelines for this purpose 2, and its use has been mandated by several publishers, including Wiley & Sons, publisher of Cytometry Part A. However, MIFlowCyt is focused mainly on wet lab components of the experiment. Although primary data along with compensation and gating details are also required to be shared, this is only sufficient for low-dimensional, low-throughput studies. With recent technological advances, data sets are growing by orders of magnitude, requiring bioinformatics tools such as those highlighted in a recent special issue in this journal 3. As analyses become more complex, so, too, does the task of ensuring that they can be reliably reproduced. What are the important details of computational analysis of FCM data that need to be captured, and how should they be provided? MIFlowCyt suggests providing the “exact mathematical formulas/algorithms” 2 for each of these, and allows for this information to be provided as text. However, MIFlowCyt is only intended to allow for understanding of the experiment (and analysis), not for reproducibility. What else is needed? This problem has received more attention in the broader biostatistics community, where reproducible research has been defined as “research papers with accompanying software tools that allow the reader to directly reproduce the results and employ the computational methods that are presented in the research article” 4. Or, put another way, in much the same way as lab protocols provided in a methods section should be sufficient to reproduce bench results, manuscripts should come with code and accompanying data which, when run, will reproduce the figures and tables in the article. Complex FCM analysis code already arouses scepticism among the less programmatically inclined in the community, for whom such analyses may be difficult to verify 5. When details of algorithmic execution are provided only in textual format, reproducing the work often requires “forensic bioinformatics” 6. This is time-consuming and requires specialized bioinformatic skills, presenting a challenge even to those with programming expertise, and an insurmountable barrier to the rest of the community. Such obstacles to reproducibility can have deleterious consequences. In the microarray community, several major biomedical studies were identified, using forensic bioinformatics, to be fundamentally flawed due to errors in the analysis 6. These erroneous results were being used to guide decision-making in clinical trials, and had direct impacts on human health. In a larger economic study, 50% of all preclinical research findings were found to be irreproducible, representing $28.2 billion in wasted funding in the USA alone 7. Given our field's close relationship with the clinic, it behooves us to avoid repeating these mistakes. Providing detailed code makes the jobs of reviewers easier, allowing for deeper and higher-quality reviews 4, and helping prevent such errors. Moving forward, how can authors make their complicated flow bioinformatics work reproducible? For starters, they should ensure that their data is readily accessible. Sharing of all research data is already required by many funding agencies, including, for example, the NIH, which states “Data sharing is essential for expedited translation of research results into knowledge, products and procedures to improve human health” 8. Requirements for data sharing are also incorporated into most journal's policies for many other data types, such as sequencing, microarrays, and protein structure, a policy that is being extended to all datatypes across top tier journals 9. For FCM data, this is easily achieved by depositing it in an open-access database, such as FlowRepository (10), which is designed specifically for FCM data, and is recommended by Wiley, PLoS, and Nature Publishing Group, among others. Next, authors must ensure that code is provided which, when executed, will reproduce all of the figures and tables in the article. A good idea to facilitate this is to ensure that the analysis uses free, open-source tools, such as the suite of packages for FCM analysis in Bioconductor that contains the vast majority of the software available for this purpose 5. Where analyses would take too long to run in a reasonable time to be part of that code, provide intermediate results of those high-throughput computations. The code used to produce those results should also be provided. Pseudocode can be useful within a manuscript to provide an overview, but this is not enough to enable reproduction or results especially where parameters are adjustable. Further, authors should make the code available under an open source license. This follows the principles of scientific research in providing a mechanism for other researchers to build upon previous work. A wide range of licenses, as well as guidelines for their applicability and use, are available 11, 12. New FCM bioinformatic methods should also be published reproducibly. When a new method is presented, it is standard practice in computational journals to require that it be compared directly to existing methods by running both the new and old methods side by side on the same data. This provides some measure of the strengths and weaknesses of the new method to guide users in choosing the appropriate method for their analysis. If all of the code and data necessary to reproduce the validation of each method is provided, the task of making such comparisons becomes significantly easier. In the FCM field, there has been a history of new methods being published, each with a lone result on a single, novel data set, but no comparison 13. In answer, the Flow Cytometry: Critical Assessment of Population Identification Methods (FlowCAP) series of competitions was established 13, 14. FlowCAP not only provides a direct comparison of existing methods, but also a series of public benchmark data sets. For types of methods already surveyed by FlowCAP, such as automated population identification, this enables easy, direct comparison to the state of the art. For methods not yet surveyed, the data sets provided by FlowCAP, along with the growing body of public data in FlowRepository 10, provide the foundations for comparison. The resources are in place to ensure that the utility and validity of every new FCM bioinformatics method can be demonstrated quantitatively, and it is imperative that we as a community make use of them. Reproducible research has received extensive attention in many other fields. Steps have begun to be taken toward its realization within the FCM community through the creation and enforcement of standards. But as both data and analyses grow ever more complex, it is essential that we embrace best practices and begin publishing code along with our research. With Cytometry A's adoption of these practices as requirements for publication, we look forward to a new era of reproducibility in FCM bioinformatics.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.