BugMentor: Generating answers to follow-up questions from software bug reports using structured information retrieval and neural text generation
Bibliographic record
Abstract
Software bug reports often lack crucial information (e.g., steps to reproduce), which makes bug resolution challenging. Developers thus ask follow-up questions to capture additional information. However, according to existing evidence, bug reporters often face difficulties answering them, which leads to the premature closing of bug reports without any resolution. Recent studies suggest follow-up questions to support the developers, but answering the follow-up questions still remains a major challenge. In this paper, we propose BugMentor, a novel approach that combines structured information retrieval and neural text generation (e.g., Mistral) to generate appropriate answers to the follow-up questions. Our technique identifies the past relevant bug reports to a given bug report, captures contextual information, and then leverages it to generate the answers. We evaluate our generated answers against the ground truth answers using four appropriate metrics, including BLEU Score and Semantic Similarity. We achieve a BLEU Score of up to 72 and Semantic Similarity of up to 92 indicating that our technique can generate understandable and good answers to the follow-up questions according to Google’s AutoML Translation documentation. Our technique also outperforms four existing baselines with a statistically significant margin. We also conduct a developer study involving 23 participants where the answers from our technique were found to be more accurate, more precise, more concise and more useful. • BugMentor combines structured retrieval and neural text generation for bug Q&A. • Incorporating the retrieved bug report context significantly improves the generated answers. • BugMentor outperforms four existing baselines. • BugMentor is compatible with multiple large language models (e.g., Llama or Mistral). • BugMentor aims to reduce developer time spent on follow-up question resolution.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.002 | 0.013 |
| Meta-epidemiology (narrow) | 0.002 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.002 | 0.001 |
| Science and technology studies | 0.000 | 0.001 |
| Scholarly communication | 0.001 | 0.002 |
| Open science | 0.002 | 0.001 |
| Research integrity | 0.002 | 0.001 |
| Insufficient payload (model declined to judge) | 0.007 | 0.003 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".