Replay-Based Continual Learning for Test Case Prioritization

In a large-scale Continuous Integration (CI) environment, regression testing can encounter high time and resource demands in ad hoc execution. Therefore, Test Case Prioritization (TCP) is crucial for enhancing the regression testing efficiency of CI. TCP methods aim to optimize regression testing by ordering test cases to effectively cover new code changes and their potential side effects and to maximize early fault detection. Traditional prioritization processes use diverse data sources, including code coverage analysis, test execution history, and domain-specific features. Heuristic-based or code-coverage-driven prioritization techniques may not be sufficient for accurate results in a rapidly changing environment. For this reason, there has been a significant shift towards employing Machine Learning (ML) techniques in TCP in recent years to harness the vast and complex datasets generated by CI practices. ML-based TCP approaches integrate multifaceted test case features from various sources to enhance the accuracy of test case prioritization. This trend reflects a broader movement towards data-driven decisionmaking in software testing, offering the potential to significantly reduce the regression testing burden by tailoring test suites more effectively to the needs of each software build, thereby saving time and resources while maintaining or improving the software quality. Recent studies have shown that the ML-based methods used in TCP can be categorized into four groups: Supervised Learning, Unsupervised Learning, Reinforcement Learning, and Natural Language Processing. Codebases for software projects can be changed rapidly by introducing new feature distributions into the CI systems. We analyzed a Java application’s CI and version control system (VCS) history data received from the International Business Machines Corporation (IBM) [1], [6], [7]. The frequent inclusion of new test suites introduced new patterns into the dataset properties. To keep up with these changes, ML models require frequent re-training on old and new datasets to maintain high accuracy on new data. The volume of the dataset tends to increase with time as more data becomes available. Frequent re-training of ML models on the entire dataset is computationally costly and requires extensive storage. Reinforcement Learning focuses on finding the best solution through reward maximization [2] and restricts the learning goal. Learning incrementally from new non-stationary data without requiring an old dataset to solve this TCP problem. Continual Learning (CL) or life-long learning/ incremental learning adapts to changes without needing old training samples. While CL has recently been studied in several works for different domains, we could not find effective research on implementing CL in the TCP domain. Given the dynamic environment of software testing, we apply CL in industrial test case prioritization is critical for maintaining the efficiency and effectiveness of software testing processes in dynamic environments. However, modifying ML models on new datasets may introduce other problems, such as catastrophic forgetting. This can occur when the model is trained on a new distribution, and the model weights change drastically. Different strategies have been suggested to solve the problem of catastrophic forgetting in CL. This abstract discusses the integration of pre-training and replay-based continual learning methods to enhance test case prioritization. Pre-trainingbased continual learning leverages the strong representation of pre-training models on a large dataset. This approach helps initialize the model with a broad understanding, which can be further incrementally trained to accommodate new tasks without significant performance loss on previous tasks. The dataset we obtained from IBM has a few years of test execution data for the CI and VCS. The model can be trained using a large volume of data for the pre-training method. Replay-based continual learning, however, involves retaining a small buffer of old training samples. This strategy includes a small fraction of old samples with a new dataset while incrementally training the model, enabling it to maintain its performance on older tasks by reinforcing the previous learning. Integrating pre-training and replay-based methods is most effective in the literature [3]. Pre-training provides a solid foundation for generic knowledge; replay-based methods complement this by continuously reinforcing past learning, ensuring that the adaptation to new tasks does not come at the expense of previously acquired knowledge. Several design choices leverage the benefits of this combined method. The frequency of incremental training on new datasets is an important design decision. This frequency can be timedriven or property-driven. Experimental work will guide the decision on incremental training frequency. Next, in replay-based approaches, the memory buffer size, number of old samples, and criteria for old sample selection are some of the decision parameters. In addition, a small buffer memory requires effective management in terms of data-retaining strategies. The empirical evidence supports the effectiveness of this integrated approach. Hu et al. [4] introduced prioritized experience replay in continual learning, emphasizing the selection of representative experiences to alleviate catastrophic forgetting. Similarly, Merlin et al. [5] provided practical recommendations for replay-based continual learning methods, highlighting the importance of memory size and data augmentation in enhancing the performance. We will conduct detailed investigations to determine the optimal values for these decision parameters. For time-based frequency, we will experiment at different intervals, such as weekly, every ten or 15 days, monthly, three months, and six months of incremental training. Property-based choices can be new test suite additions, significant changes in test suites, and an increase or decrease in the test case failure rate. Similarly, for the replay-based method, samples can be selected from each incremental training dataset; the selection can be random or property-based. For example, an even distribution of passed or failed samples could be selected to avoid overfitting. In conclusion, integrating pretraining and replay-based continual learning methods presents a promising research direction for enhancing large-scale test case prioritization in CI. Future research should explore different strategies to maximize the benefits of continual learning in test case prioritization.

Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.