Mobile Application Coverage: The 30% Curse and Ways Forward
Bibliographic record
Abstract
Abstract Testing, security analysis, and other dynamic quality assurance approaches rely on mechanisms that invoke the software under test, aiming to achieve high code coverage. A large number of invocation mechanisms proposed in the literature, in particular for Android mobile applications, employ GUI-driven application exploration. However, studies show that even the most advanced GUI exploration techniques can cover only around 30% of a real-world application. This paper aims to investigate “the remaining 70%”. By conducting a large-scale experiment involving two human experts, who thoroughly explored 61 benchmark and 42 popular apps from Google Play, we show that achieving a substantially larger coverage for real-world applications is impractical even if we factor out known GUI-based exploration issues, such as the inability to provide semantic inputs and the right order of events. The main reasons preventing even human analysts from covering the entire application include application dependencies on remote servers and external resources, hard-to-reach app entry points, disabled and erroneous features, and software/hardware properties of the underlying device. Thus, future investment in GUI-based exploration strategies is unlikely to lead to substantial improvements in coverage. To chart possible ways forward and explore approaches to satisfy/bypass these “blockers”, we thoroughly analyze code-level properties guarding them. Our analysis shows that a large fraction of the blockers could actually be successfully bypassed with relatively simple beyond-GUI exploration techniques. We hope our study can inspire future work in this area; it also provides a realistic benchmark for evaluating such work. Purpose In this artifact, we provide our application exploration and coverage analysis results, in an effort to inspire the community to invest in coverage explainability and application-specific exploration strategies that go beyond GUI exploration. We include detailed information about our evaluation benchmarks for tool and manual exploration, e.g., package name, version used, source, category, downloads, SDK levels, permissions, etc. We also report coverage results achieved by tools and human analysts with plots of the coverage progression over time for analysts as well as the scripts used for the coverage experiments. We further provide manual analysis results for our case study, more specifically extracted reasons for unreachability of case study apps and associated code-level properties guarding unreached portions of the applications. Finally, we identify a list of beyond-GUI exploration techniques and categorize them for future work to reference and build upon when designing novel app exploration approaches. We hope for our results to serve as a precursor to building automated techniques which can identify and explain unreachable portions of an application from code, to constitute a ground truth for such techniques, and ultimately better orient testing/exploration efforts. Data from the Mobile Coverage Project, by Faridah Akinotcho, Lili Wei and Julia Rubin is made available under the MIT License: https://opensource.org/license/mit
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.016 | 0.125 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.005 | 0.002 |
| Science and technology studies | 0.002 | 0.003 |
| Scholarly communication | 0.005 | 0.007 |
| Open science | 0.002 | 0.005 |
| Research integrity | 0.002 | 0.004 |
| Insufficient payload (model declined to judge) | 0.003 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".