Evaluating the Use of Semantics for Identifying Task-relevant Textual Information
Bibliographic record
Abstract
The information a developer seeks to aid the completion of a task typically exists across a range of artifacts. For example, for a task that requires upgrading to a new version of an API component, a developer may seek information in the API's official documentation, check community discussions about the newer version, and so on. To aid developers in locating the portion of the text that might be useful in these artifacts, prior work has used syntactic properties of the text, and an artifact's meta-data, to automatically identify relevant text for particular kinds of artifacts. Although effective, these techniques rely on assumptions about an artifact's structure or content that prevent applying them across the different types of artifacts that a developer may come across in their daily work. In this paper, we investigate whether techniques building on approaches to interpret the meaning, or semantics, of the text help to overcome these limitations. Particularly, we introduce six semantic-based techniques and evaluate that they can identify up to 58% of the text that developers deem relevant to Android development tasks. When compared to a state-of-the-art approach, we find that our techniques achieve comparable recall values, identifying 63% of the small fraction of the task-relevant text of Stack Overflow artifacts, but without the need for artifact-specific information.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.010 | 0.067 |
| Meta-epidemiology (narrow) | 0.002 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.012 | 0.003 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.002 | 0.005 |
| Open science | 0.001 | 0.002 |
| Research integrity | 0.002 | 0.001 |
| Insufficient payload (model declined to judge) | 0.001 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".