ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolutionary Search – Replication Package

This is the replication package associated with the paper "<em>ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolutionary Search</em>" accepted at the 45th IEEE/ACM International Conference on Software Engineering (ICSE 2023) – Technical Track. Cite this paper using the following: <em>@inproceedings{pan2023atm,<br> title={ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolutionary Search},<br> author={Pan, Rongqi and Ghaleb, Taher A. and Briand, Lionel},<br> booktitle={Proceedings of the 45th IEEE/ACM International Conference on Software Engineering},<br> year={2023},<br> pages={1--12}<br> }</em> <strong>Replication Package Contents:</strong><br> The replication package contains all the necessary data and code required to reproduce the results reported in the paper. We also provide the results for other minimization budgets, and detailed <em>FDR,</em> execution time, and statistical test results. In addition, we provide the data and code required to reproduce the results of baselines techniques: FAST-R and random minimization. <strong>Data:</strong><br> We provide in the <em><strong>Data</strong></em> directory the data used in our experiments, which is based on 16 projects from Defects4J, whose characteristics can be found in <em><strong>Data/subject_projects.csv</strong></em><em>.</em> <strong>Code:</strong><br> We provide in the <em><strong>Code</strong></em> directory the code and scripts (Java, Python, and Bash) required to run the experiments and reproduce the results. <strong>Results:</strong><br> We provide in the <em><strong>Results</strong></em> directory the results for each technique independently, and also a summary of all results together for comparison purposes. The source code for this step is in the <em><strong>Code/ATM/CodeToAST</strong></em> directory. The source code for this step is in the <em><strong>Code/ATM/Similarity</strong></em> directory. <strong>_________________________________</strong> <strong>ATM - Code to AST transformation:</strong> <strong>Requirements:</strong><br> * Eclipse IDE (we used 2021-12)<br> * The libraries (the <em><strong>.jar</strong></em> files in the <em><strong>Code/ATM/CodeToAST/lib</strong></em> directory) <strong>Input:</strong><br> All zipped data files should be unzipped before running each step.<br> * Data/test_suites/all_test_cases.zip → Data/test_suites/all_test_cases<br> * Data/test_suites/changed_test_cases.zip → Data/test_suites/changed_test_cases<br> * Data/test_suites/relevant_test_cases.zip → Data/test_suites/relevant_test_cases <strong>Output:</strong><br> * Data/ATM/ASTs/all_test_cases<br> * Data/ATM/ASTs/changed_test_cases <strong>Running the experiment:</strong><br> To generate ASTS for all test cases in the project test suites, the <em><strong>Code/ATM/CodeToAST/src/CodeToAST.java</strong></em> file should be compiled and run using the Eclipse IDE by including all the required <em><strong>.jar</strong></em> files in the <em><strong>Code/ATM/CodeToAST/lib</strong></em> directory as part of the classpath. A bash script is provided along with a pre-generated <em><strong>.jar</strong></em> file in the <em><strong>Code/ATM/CodeToAST/bin</strong></em> directory to run this step, as follows: <pre><code class="language-bash">cd Code/ATM/CodeToAST bash transform_code_to_ast.sh</code></pre> Each test file in the <em><strong>Data/test_suites/all_test_cases</strong></em> and <em><strong>Data/test_suites/changed_test_cases</strong></em> directories is parsed to generate a corresponding AST for each test case method (saved in an XML format in <strong>Data/ATM/ASTs/all_test_cases</strong> and <em><strong>Data/ATM/ASTs/changed_test_cases</strong></em> for each project version)<br> <strong>_________________________________</strong> <strong>ATM - Similarity Measurement:</strong> <strong>Requirements:</strong><br> * Eclipse IDE (we used 2021-12)<br> * The libraries (the <em><strong>.jar</strong></em><strong> </strong>files in the <em><strong>Code/ATM/Similarity/lib</strong></em> directory)<br> <br> <strong>Input:</strong><br> * Data/test_suites/all_test_cases<br> * Data/test_suites/changed_test_cases<br> <br> <strong>Output:</strong><br> * Data/ATM/similarity_measurements<br> <br> <strong>Running the experiment:</strong><br> To measure the similarity between each pair of test cases, the <em><strong>Code/ATM/Similarity/src/SimilarityMeasurement.java</strong></em> file should be compiled and run using the Eclipse IDE by including all the required <em><strong>.jar</strong></em> files in the <em><strong>Code/ATM/Similarity/lib</strong></em> directory as part of the classpath. A bash script is provided along with a pre-generated <em><strong>.jar</strong></em> file in the <em><strong>Code/ATM/Similarity/bin</strong></em> directory to run this step, as follows: <pre><code class="language-bash">cd Code/ATM/Similarity bash measure_similarity.sh</code></pre> ASTs of each project in the <em><strong>Data/ATM/ASTs/all_test_cases</strong></em> and <em><strong>Data/ATM/ASTs/changed_test_cases</strong></em> directories are parsed to create pairs of ASTs containing one test case from the <em><strong>Data/ATM/ASTs/all_test_cases</strong></em> directory with another test case from the <em><strong>Data/ATM/ASTs/changed_test_cases</strong></em> directory (redundant pairs are discarded). Then, all similarity measurements are saved in the <em><strong>Data/ATM/similarity_measurements.zip</strong></em> file.<br> __________________________________________ <strong>Search-based Minimization Algorithms:</strong><br> The source code for this step is in the <em><strong>Code/ATM/Search</strong></em> directory.<br> <br> <strong>Requirements:</strong><br> To run this step, Python 3 is required (we used <em><strong>Python 3.10</strong></em>). Also, the libraries in the <strong>Code/AMT/Search/requirements.txt</strong> file should be installed, as follows: <pre><code class="language-bash">cd Code/ATM/Search pip install -r requirements.txt</code></pre> <strong>Input:</strong><br> * Data/ATM/similarity_measurements<br> <br> <strong>Output:</strong><br> * Results/ATM/minimization_results<br> <br> <strong>Running the experiment:</strong><br> To minimize the test suites in our dataset, the following bash script should be executed: <pre><code class="language-bash">bash minimize.sh</code></pre> All similarity measurements are parsed for each version of the projects, independently. Each version is run 10 times using three minimization budgets (25%, 50%, and 75%). Genetic Algorithm (GA) is run using four similarity measures, namely top-down, bottom-up, combined, and tree edit distance. NSGA-II is run using two combinations of similarity measures: top-down &amp; bottom-up and combined &amp; tree edit distance. The minimization results are generated in the <em><strong>Results/ATM/minimization_results</strong></em> directory.<br> __________________ <strong>Evaluate results:</strong><br> To evaluate and summarize the minimization results, run the following: <pre><code class="language-bash">cd Code/ATM/Evaluation bash evaluate.sh</code></pre> This will generate summarized <em>FDR</em> and execution time results (per-project and per-version) for each minimization budget, which can all be found in <strong>Results/ATM</strong>. In this replication package, we provide the final, merged <em>FDR</em> with execution time results. <strong>_________________________________</strong> <strong>Running FAST-R experiments</strong><br> ATM was compared to FAST-R, a state-of-the-art baseline, which is a set of test case minimization techniques called: <em>FAST++, FAST-CS, FAST-pw, and FAST-all</em>, which we adapted to our data and experimental setup. <strong>Requirements:</strong><br> To run this step, Python 3.7 is required. Also, the libraries in the <em><strong>Code/FAST-R/requirements.txt</strong></em> file should be installed, as follows: <pre><code class="language-bash">cd Code/FAST-R pip install -r requirements.txt</code></pre> <strong>Input:</strong><br> * Data/FAST-R/test_methods<br> * Data/FAST-R/test_classes <strong>Output:</strong><br> * Results/FAST-R/test_methods/FDR_and_Exec_Time_Results_[budget]%_budget.csv<br> * Results/FAST-R/test_classes/FDR_and_Exec_Time_Results_[budget]%_budget.csv<br> <br> To run FAST-R experiments, the following bash script should be executed: <pre><code class="language-bash">bash fast_r.sh test_methods #method level bash fast_r.sh test_classes #class level</code></pre> Results are generated in <em><strong>.csv</strong></em> files for each budget. For example, for the 50% budget, results are saved in <strong>FDR_and_Exec_Time_Results_50%_budget.csv</strong> in the <em><strong>Results/FAST-R/test_methods</strong></em> and <em><strong>Results/FAST-R/test_classes</strong></em> directories. <strong>_________________________________</strong> <strong>Running the random minimization experiments</strong><br> ATM was also compared to random minimization as a standard baseline. <strong>Requirements:</strong> To run this step, Python 3 is required (we used <em><strong>Python 3.10</strong></em>). Also, the libraries in the <em><strong>Code/RandomMinimization/requirements.txt</strong></em> file should be installed, as follows: <pre><code class="language-bash">cd Code/RandomMinimization pip install -r requirements.txt</code></pre> <strong>Input:</strong><br> <em>N/A</em> <strong>Output:</strong><br> * Results/RandomMinimization/FDR_and_Exec_Time_Results_[budget]%_budget.csv<br> <br> To run the random selection experiments, the following bash script should be executed: <pre><code class="language-bash">bash random_minimization.sh</code></pre> Results are generated in <em><strong>.csv</strong></em> files for each budget. For example, for the 50% budget, results are saved in <em><strong>FDR_and_Exec_Time_Results_50%_budget.csv</strong></em> in the <em><strong>Results/RandomMinimization</strong></em> directory.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.