Flakify: A Black-Box, Language Model-based Predictor for Flaky Tests – Replication Package
Pourquoi ce travail est dans la base
Une base qui oublie comment elle a trouvé un travail ne peut pas être vérifiée. Voici les voies qui ont admis celui-ci.
Notice bibliographique
Résumé
This is the replication package associated with the paper: <em>Flakify: A Black-Box, Language Model-based Predictor for Flaky Tests.</em> We explain how to use it to reproduce the results reported in the paper. A maintainable version of this replication package is available on GitHub (https://github.com/uOttawa-Nanda-Lab/Flakify). <strong>Flakify Test Smell Detector</strong> This is a step-by-step guideline to detect test smells in the source code of test cases and retain statements that match them. <em><strong>Requirements:</strong></em> Eclipse IDE (the version we used was 2021-12) The libraries (the <strong><em>.jar</em></strong> files in the <strong><code>lib\</code></strong> directory) <em><strong>Input Files:</strong></em> This is a list of input files that are required to accomplish this step: <em>dataset/FlakeFlagger/FlakeFlagger_filtered_dataset.csv</em> <em>dataset/FlakeFlagger/FlakeFlagger_class_files/</em> <em>dataset/IDoFT/IDoFT_filtered_dataset.csv</em> <em>dataset/IDoFT/IDoFT_class_files/</em> The <strong><code>dataset/FlakeFlagger/FlakeFlagger_filtered_dataset.csv</code></strong> and <strong><code>dataset/IDoFT/IDoFT_filtered_dataset.csv</code></strong> are used to obtain the label (<em>flaky</em>=1 or <em>non-flaky</em>=0) and project name for each test case parsed from <strong><code>dataset/FlakeFlagger/FlakeFlagger_class_files/</code></strong> and <strong><code>dataset/IDoFT/IDoFT_class_files/</code></strong>, respectively. <strong><em>Output Files:</em></strong> <em>dataset/FlakeFlagger/FlakeFlagger_dataset.csv</em> <em>dataset/FlakeFlagger/FlakeFlagger_test_cases_full_code/</em> <em>dataset/FlakeFlagger/FlakeFlagger_test_cases_preprocessed_code/</em> <em>dataset/IDoFT/IDoFT_dataset.csv</em> <em>dataset/IDoFT/IDoFT_test_cases_full_code/</em> <em>dataset/IDoFT/IDoFT_test_cases_preprocessed_code/</em> <strong>Replicating the experiment</strong> To detect test smells and retain only code statements related to them, the <strong><code>src/FlakifySmellsDetector.java</code></strong> file should be compiled and run using the Eclipse IDE by having all the <em>.jar</em> files in the classpath. The pre-generated executable Jar file <strong><code>src/FlakifySmellsDetector.jar</code></strong> can be executed using the shell script <strong><code>src/FlakifySmellsDetector.sh</code></strong> after changing paths for each dataset as needed, using the following commands: <pre><code class="language-bash">bash FlakifySmellsDetector.sh FlakeFlagger bash FlakifySmellsDetector.sh IDoFT</code></pre> It will generate the dataset required to run Flakify's flaky test prediction model for the datasets given as input. The class file containing each of the test cases is then parsed to produce the corresponding full code and pre-processed code of the test case. The full and pre-processed source code of all test cases are also combined and saved in a CSV file, along with test smells found, project names, and labels. <strong>Flakify Replication</strong> This is the guideline for replicating the experiments we used to evaluate Flakify for classifying test cases as <em>flaky</em> and <em>non-flaky</em> using both cross-validation and per-project validation. <em><strong>Requirements:</strong></em> This is a list of all required python packages: <em>python =3.8.5</em> <em>imbalanced_learn= 0.8.1</em> <em>numpy= 1.19.5</em> <em>pandas= 1.3.3</em> <em>transformer= 4.10.2</em> <em>torch=1.5.0</em> <em>scikit_learn= 0.22.1</em> <em><strong>Input Files:</strong></em> This is a list of input files that are required to accomplish this step: <em>dataset/FlakeFlagger/Flakify_FlakeFlagger_dataset.csv</em> <em>dataset/IDoFT/Flakify_IDoFT_dataset.csv</em> This file contains the full code and pre-processed code of the test cases in both FlakeFlagger and IDOFT datasets, along with their ground truth labels (<em>flaky</em> and <em>non-flaky</em>). <em><strong>Output File:</strong></em> <em>results/Flakify_cross_validation_results_on_FlakeFlagger_dataset.csv</em> <em>results/Flakify_per_project_results_on_FlakeFlagger_dataset.csv</em> <em>results/Flakify_model_trained_on_FlakeFlagger_dataset.pt</em> <em>results/Flakify_cross_validation_results_on_IDoFT_dataset.csv</em> <em>results/Flakify_per_project_results_on_IDoFT_dataset.csv</em> <em>results/Flakify_model_trained_on_IDoFT_dataset.pt</em> <strong>Replicating Flakify experiments</strong> <strong>Cross-Validation</strong> To run the Flakify experiment using cross-validation on the two datasets, navigate to <code>src\</code> folder and run the following commands: <pre><code class="language-bash">bash Flakify_predictor_cross_validation.sh FlakeFlagger bash Flakify_predictor_cross_validation.sh IDoFT</code></pre> This will generate the classification results into <strong><code>results/Flakify_cross_validation_results_on_FlakeFlagger_dataset.csv</code></strong> and <strong><code>results/Flakify_cross_validation_results_on_IDoFT_dataset.csv</code></strong> for the cross-validation experiments on both datasets. It will also save the weights of the two models trained on the FlakeFlagger and IDoFT datasets into <strong><code>results/Flakify_model_trained_on_FlakeFlagger_dataset.pt</code></strong> and <code><strong>results/Flakify_model_trained_on_IDoFT_dataset.pt</strong></code>, respectively. <strong>Per-project Validation</strong> To run the Flakify experiment using per-project validation on the two datasets, navigate to <code>src\</code> folder and run the following commands: <pre><code class="language-bash">bash Flakify_predictor_per_project.sh FlakeFlagger bash Flakify_predictor_per_project.sh IDoFT</code></pre> This will generate the classification results into <strong><code>results/Flakify_per_project_results_on_FlakeFlagger_dataset.csv</code></strong> and <strong><code>results/Flakify_per_project_results_on_IDoFT_dataset.csv</code></strong> for the whole per-project validation experiments on both datasets. <strong>FlakeFlagger Replication</strong> This is the guideline for replicating the experiments we used to evaluate the two versions of FlakeFlagger, white-box and black-box, for classifying test cases as <em>flaky</em> and <em>non-flaky</em> using cross-validation on the FlakeFlagger dataset. <em><strong>Requirements:</strong></em> This is a list of all required python packages: <em>python =3.8.5</em> <em>imbalanced_learn= 0.8.1</em> <em>pandas= 1.3.3</em> <em>scikit_learn= 0.22.1</em> <em><strong>Input File:</strong></em> This is a list of input files that are required to accomplish this step: <em>dataset/FlakeFlagger/FlakeFlagger_filtered_dataset.csv</em> <em>dataset/FlakeFlagger/FlakeFlaggerFeaturesTypes.csv</em> <em>dataset/FlakeFlagger/Information_gain_per_feature.csv</em> <em><strong>Output File:</strong></em> <em>results/FlakeFlagger_black-box_results.csv</em> <em>results/FlakeFlagger_white-box_results.csv</em> <strong>Replicating FlakeFlagger experiments</strong> To run the FlakeFlagger experiments, navigate to <code>src\</code> folder and run the following command: <pre><code class="language-bash">bash FlakeFlagger_predictor.sh white-box bash FlakeFlagger_predictor.sh black-box</code></pre> This will generate the classification results into <strong><code>results/FlakeFlagger_white-box_results.csv</code> </strong>and <strong><code>results/FlakeFlagger_black-box_results.csv</code> </strong>for both white-box and black-box experiments, respectively.
Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.
Prédiction distillée sur la base complète
Imitation des enseignantsNi prévalence calibrée, ni vérité terrain. Validation humaine à venir. Apprise à partir de 10 348 étiquettes directes de Codex et de 10 348 étiquettes directes de Gemma. Le mode candidate est l'union des têtes enseignantes seuillées; le consensus est leur intersection. Ces sorties portent le statut machine_predicted_unvalidated et ne sont ni des étiquettes humaines ni des étiquettes directes de modèles de pointe.
Scores Codex et Gemma par catégorie
| Catégorie | Codex | Gemma |
|---|---|---|
| Métarecherche | 0,001 | 0,005 |
| Méta-épidémiologie (sens strict) | 0,000 | 0,000 |
| Méta-épidémiologie (sens large) | 0,000 | 0,000 |
| Bibliométrie | 0,001 | 0,001 |
| Études des sciences et des technologies | 0,002 | 0,000 |
| Communication savante | 0,001 | 0,000 |
| Science ouverte | 0,004 | 0,003 |
| Intégrité de la recherche | 0,000 | 0,001 |
| Charge utile insuffisante (le modèle a refusé de juger) | 0,004 | 0,001 |
Scores machine (provisoires)
Les deux têtes enseignantes du modèle étudiant, lues sur ce travail. Un score ordonne la base pour la relecture; il n'affirme jamais une catégorie, et le statut de validation accompagne chaque rangée tel quel.
Scores de référence d'un modèle non mature (critères de maturité non atteints, 7 itérations). Un score ordonne; il n'affirme jamais une catégorie.
score_only:v0-immature-baseline · tel quel depuis la passe de notation : score_only signifie que le nombre peut ordonner les travaux, et qu'aucune étiquette de catégorie n'en découle