A Structured Testing Framework for ADAS Software Development
Bibliographic record
Abstract
A major task in the design of automated vehicles is the need to quickly and thoroughly validate a development teams algorithms. There currently exists no explicitly defined common standard for developers working on Advanced Driver Assisted Systems to adopt during their software testing process. Instead different teams customize their testing process specifically to their software systems current needs. Literature indicates that these processes can be comprehensive but convoluted, and not flexible to change as test requirements and the system itself does. This thesis introduces a test framework at the unit, integration, and system test levels with the objective of addressing these challenges through a complete test framework centered around rapid execution and modular test design. At the unit test level a recommendation guide is put forth that is largely aimed at new developers with concrete actionable items that can be integrated into a teams process. For integration and system level testing, a software solution for ROS based development referred to as University of Waterloo Structured Testing Framework (UW-STF) is described in regards to both the benefits it provides as well as its low level implementation details. This includes how to tie the framework into using data generated from the popular simulator CARLA for end-to-end testing of a system. Lastly the test framework is applied to the codebase of UWAFT for their development efforts related to connected and automated vehicles. The framework was shown to increase readability/clarity at the unit test level, facilitate robust automated testing at the integration level and provide transparency on the teams current algorithms performance at the system test level (average F1-score of 0.77 and average OSPA of 2.42). When compared to the standard ROS integration test framework, UW-STF executed the same test suite with 60%+ reduction in lines of code and meaningful differences in CPU and memory requirements.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.021 | 0.030 |
| Meta-epidemiology (narrow) | 0.002 | 0.002 |
| Meta-epidemiology (broad) | 0.001 | 0.004 |
| Bibliometrics | 0.005 | 0.002 |
| Science and technology studies | 0.002 | 0.005 |
| Scholarly communication | 0.007 | 0.005 |
| Open science | 0.006 | 0.005 |
| Research integrity | 0.003 | 0.006 |
| Insufficient payload (model declined to judge) | 0.010 | 0.007 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".