LTM: Scalable and Black-box Similarity-based Test Suite Minimization based on Language Models - Replication Package
Bibliographic record
Abstract
LTM: Scalable and Black-box Similarity-based Test Suite Minimization based on Language Models This is the replication package associated with the paper "LTM: Scalable and Black-box Similarity-based Test Suite Minimization based on Language Models". Replication Package Contents: This replication package contains all the necessary data and code required to reproduce the results reported in the paper. We provide the results of the Fault Detection Rate (FDR), Total Minimization Time (MT), Time Saving Rate (TSR) , statistical tests for all the minimization budgets (i.e., 25%, 50%, and 75%), results for the preliminary study, results for UniXcoder/Cosine with preprocessed code on 16 projects. Data: We provide in the Data directory the data used in our experiments, which is the source code of test cases (Java test methods) of 17 projects collected from Defects4J. Code: We provide in the Code directory the code (Python) and bash files required to run the experiments and reproduce the results. Results: We provide in the Results directory the detailed results for our approach (called LTM). We also provide the summarized results of LTM and a baseline (ATM) for comparison purposes. Additional technical details about ATM can be found at https://zenodo.org/record/7455766. _________________________________ LTM's Similarity Measurement: The source code of this step is in the Code/LTM/Similarity directory. Requirements: To run this step, Python 3 is required (we used Python 3.10). Also, the required libraries in the Code/LTM/Similarity/requirements.txt file should be installed, as follows: cd Code/LTM/Similarity pip install -r requirements.txt Input: Data/LTM/TestMethods Output: Data/LTM/similarity_measurements Running the experiment: To measure the similarity between all pairs of test cases, the following bash script should be executed: bash measure_similarity.sh The source code of test methods of each project in the Data/LTM/TestMethods is parsed to generate pairs of test cases. This steps includes test methods tokenization, test methods embeddings extraction and similarity calculation. Then, all similarity scores are stored in Data/LTM/similarity_measurements folder. Due to the large size of the calculated similarity scores (60 GB), they were not uploaded on Zenodo, but they can be available upon request. LTM's Test Suite Minimization: The source code of this step is in the Code/LTM/Search directory. Requirements: To run this step, Python 3 is required (we used Python 3.10). Also, the required libraries in the Code/LTM/Search/requirements.txt file should be installed, as follows: cd Code/LTM/Search pip install -r requirements.txt Input: Data/LTM/similarity_measurements Output: Results/LTM/minimization_results Running the experiments: To minimize the test suite for each project version, the following bash script should be executed: bash minimize.sh The similarity scores of all test case pairs per project version are parsed by the search algorithm (Genetic Algorithm). Each experiment runs ten times using three minimization budgets (25%, 50%, and 75%). The results are stored in the Results/LTM/minimization_results directory. LTM's Evaluation: To evaluate the minimization results for each version and each project, the following bash script should be executed: cd Code/LTM/Evaluation bash evaluate_per_version.sh cd Code/LTM/Evaluation bash evaluate_per_project.sh This will evaluate the FDR, MT and TSR results for each version and each project for each minimization budget. These results are stored in the Results/LTM directory. Note that for each version, the FDR is either 1 or 0. For each project, the FDR ranges from 0 to 1.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.005 | 0.028 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.003 |
| Bibliometrics | 0.002 | 0.001 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.002 | 0.003 |
| Open science | 0.005 | 0.004 |
| Research integrity | 0.001 | 0.003 |
| Insufficient payload (model declined to judge) | 0.026 | 0.014 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".