Studying the Impact of Noises in Build Breakage Data [Online Appendix]
Bibliographic record
Abstract
This document represents an Online Appendix of our article, which has been accepted for publication in the IEEE Transactions on Software Engineering (TSE) in August 2019. The Online Appendix includes a replication package of the scripts and data we used in our work, in addition to more detailed results of the findings reported in the article. <strong>Abstract.</strong> Much research has investigated the common reasons for build breakages. However, prior research has paid little attention to builds that may break due to reasons that are unlikely to be related to development activities. For example, Continuous Integration (CI) builds may break due to timeout or connection errors while generating the build. Such kinds of build breakages potentially introduce noises to build breakage data. Not considering such noises may lead to misleading results when studying CI builds. In this paper, we propose three criteria to identify build breakages that can potentially introduce noises to build breakage data. We apply these criteria to a dataset of 350,246 builds from 153 GitHub projects that are linked with Travis CI. Our results reveal that 33% of the build breakages are due to environmental factors (e.g., errors in CI servers), 29% are due to (unfixed) errors in previous builds, and 9% are due to build jobs that were later deemed by developers as noisy (there is an overlap of 17% between these three types of breakages). We measure the impact of noises in build breakage data on modeling build breakages. We observe that models that use uncleaned build breakage data can lead to misleading associations between build breakages and development activities (e.g., the role of developer). However, such associations could not be observed after eliminating noisy build breakages. Moreover, we replicate a prior study that investigates the association between build breakages and development activities using data from 14 GitHub projects. We observe that some observations reported by the prior study (e.g., pull requests cause more breakages) do not hold after eliminating the noises from build breakage data. <strong>Citing this research.</strong> If you intend to use any materials (i.e., scripts, data, approach, or findings) of this work, please cite our article as follows: @article{ghaleb2019studying,<br> title={Studying the Impact of Noises in Build Breakage Data},<br> author={Ghaleb, Taher Ahmed and da Costa, Daniel Alencar and Zou, Ying and Hassan, Ahmed E.},<br> journal={IEEE Transactions on Software Engineering},<br> pages={1--14},<br> year={2019},<br> publisher={IEEE},<br> DOI={10.1109/TSE.2019.2941880}<br> } The official publication of this research can be found at https://dx.doi.org/10.1109/TSE.2019.2941880 <strong>Replication Package.</strong> You may download the whole replication package (i.e., scripts, data, approach, or findings) using the zip file shown below. Please note that the size of the raw build logs used in this work is approximately 107 GB. You can download raw build logs using from Travis CI (using Travis API) or its AWS S3 backend (using Amazon S3 API).
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.000 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.002 | 0.000 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.023 | 0.002 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; both teacher heads agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".