Putting benchmarks in their rightful place: The heart of computational biology

Research in computational biology has given rise to a vast number of methods developed to solve scientific problems.For areas in which many approaches exist, researchers have a hard time deciding which tool to select to address a scientific challenge, as essentially all publications introducing a new method will claim better performance than all others.Not all of these claims can be correct.Equally, for this same reason, developers struggle to demonstrate convincingly that they created a new and superior algorithm or implementation.Moreover, the developer community often has difficulty discerning which new approaches constitute true scientific advances for the field.The obvious answer to this conundrum is to develop benchmarks-meaning standard points of reference that facilitate evaluating the performance of different tools-allowing both users and developers to compare multiple tools in an unbiased fashion.Broadly speaking, benchmarks consist of input data that methods are meant to operate upon, expected output data against which tool output can be compared, a specification of metrics used to assess performance, and performance values of sets of tools that have been run through the benchmark.Developing good and comprehensive benchmarks, in which the performance metrics of each tool reflect its real-world utility, requires a significant effort.For highly competitive and established fields, such as protein structure predictions, community experiments evaluating the methods have been held periodically to provide blinded assessments of prediction performance.These blinded assessments are perhaps the gold standard on how benchmarks should be run.However, in most areas of computational biology, no such regular blinded contests are available.Instead, many tool developers end up generating their own benchmarks, which they publish alongside a newly developed tool to show its improved performance.The downside of this approach is that, if a new approach is developed in parallel to assembly of the benchmark on which it is evaluated, there is a strong selection bias encouraging the authors to report tool development approaches performing well against the benchmark compared to previous tools.This reporting bias makes most benchmarks that accompany newly developed tools questionable.Even if the authors are aware of this problem and take conscious steps to separate

Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.