Predictive Power of Large Language Models in Bug Severity Detection: An Explorative Study
Bibliographic record
Abstract
RÉSUMÉ: Les systèmes de suivi des bogues sont essentiels en génie logiciel car ils facilitent la gestion structurée des défauts et des améliorations logicielles. Ces systèmes servent d’outils pour surveiller et traiter les problèmes survenant lors des phases de développement et de maintenance des applications logicielles. Les utilisateurs utilisent les systèmes de suivi des bogues pour soumettre des rapports détaillés, décrivant les problèmes rencontrés et leur impact potentiel sur l’expérience utilisateur. Une fois soumis, ces rapports passent par une phase cruciale appelée triage des bogues, où une équipe d’experts évalue systématiquement la sévérité, la priorité et l’impact global de chaque problème signalé. Cependant, le processus d’évaluation de la sévérité des bogues présente des défis en raison de sa nature subjective. La variabilité dans l’évaluation de la sévérité est influencée par les perceptions individuelles, les expériences et les facteurs contextuels, ce qui complique davantage le processus de triage et de résolution des bogues. De plus, le volume important de rapports de bogues générés quotidiennement au sein des plates-formes de suivi des bogues souligne le besoin pressant d’automatiser les décisions de sévérité. Bien que des recherches antérieures aient exploré l’automatisation de la prédiction de la sévérité des bogues en utilisant des techniques de traitement du langage naturel (par exemple, la tokenisation, la suppression des mots vides, le stemming et la lemmatisation) et des algorithmes d’apprentissage machine traditionnels (par exemple, Naive Bayes, k-Nearest Neighbors et Arbres de décision), des progrès récents dans l’apprentissage en profondeur, notamment les modèles basés sur des transformateurs tels que BERT, Llama et GPT, offrent le potentiel de renforcer considérablement la précision de prédiction. Ces modèles subissent un pré-entraînement approfondi sur des données textuelles, ce qui leur permet de capturer des motifs complexes et des nuances contextuelles inhérentes au langage. Contrairement aux embeddings algorithmiques conventionnels, qui peuvent avoir du mal à capturer des relations sémantiques subtiles, les grands modèles de langage excellent dans la génération d’embeddings qui reflètent adéquatement le sens sémantique et les nuances contextuelles des mots dans les phrases et les documents. ABSTRACT: Bug tracking systems are essential in software engineering by facilitating the structured management of software defects and enhancements. These systems serve as tools for monitoring and addressing issues that arise during the development and maintenance phases of software applications. Users leverage bug tracking systems to submit detailed reports, outlining encountered issues and their potential impact on user experience. Once submitted, these reports undergo a crucial phase called bug triage, wherein a team of experts systematically evaluates the severity, priority, and overall impact of each reported issue. However, the process of assessing bug severity inherently presents challenges due to its subjective nature. The variability in severity assessment is influenced by individual perceptions, experiences, and contextual factors, further complicating the bug triage and resolution process. Moreover, the sheer volume of bug reports generated daily within bug tracking systems platforms highlights the pressing need for automating severity decisions. Although previous research has explored automating bug severity prediction using Natural Language Processing (e.g. tokenization, stop words removal, stemming, and lemmatization) techniques and traditional machine learning algorithms (e.g. Naive Bayes, k-Nearest Neighbors, and Decision Trees), recent strides in deep learning, notably transformer-based models such as BERT, Llama, and GPT, offer the potential to reinforce prediction accuracy significantly. These models undergo extensive pre-training on textual data, enabling them to capture intricate patterns and contextual nuances inherent in language. Unlike conventional algorithmic embeddings, which may struggle to capture subtle semantic relationships, large language models excel at generating contextual embeddings that adequately reflect the semantic meaning and contextual nuances of words within sentences and documents.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.013 | 0.060 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.002 |
| Bibliometrics | 0.002 | 0.002 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.003 | 0.003 |
| Open science | 0.001 | 0.001 |
| Research integrity | 0.001 | 0.002 |
| Insufficient payload (model declined to judge) | 0.002 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".