Test Re-Prioritization in Continuous Testing Environments
Bibliographic record
Abstract
New changes are constantly and concurrently being made to large software systems. In modern continuous integration and deployment environments, each change requires a set of tests to be run. This volume of tests leads to multiple test requests being made simultaneously, which warrant prioritization of such requests. Previous work on test prioritization schedules queued tests at set time intervals. However, after a test has been scheduled it will never be reprioritized even if new higher risk tests arrive. Furthermore, as each test finishes, new information is available which could be used to reprioritize tests. In this work, we use the conditional failure probability among tests to reprioritize tests after each test run. This means that tests can be reprioritized hundreds of times as they wait to be run. Our approach is scalable because we do not depend on static analysis or coverage measures and simply prioritize tests based on their co-failure probability distributions. We named this approach CODYNAQ and in particular, we propose three prioritization variants called CODYNAQSINGLE, CODYNAQDOUBLE and CODYNAQFLEXI. We evaluate our approach on two data sets, CHROME and Google testing data. We find that our co-failure dynamic re-prioritization approach, CODYNAQ, outperforms the default order, FIFOBASELINE, finding the first failure and all failures for a change request by 31% and 62% faster, respectively. CODYNAQ also outperforms GOOGLETCP by finding the first failure 27% faster and all failures 62% faster.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.007 | 0.024 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.003 | 0.002 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.002 | 0.003 |
| Open science | 0.004 | 0.002 |
| Research integrity | 0.001 | 0.002 |
| Insufficient payload (model declined to judge) | 0.002 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".