An Industrial Study on Predicting Crash Report Log \nTypes Using Large Language Models
Bibliographic record
Abstract
Software crashes and failures take a fair amount of effort and time to resolve. Software developers \nuse information submitted in crash reports (CRs) to conduct root cause analysis of faults. The \nproblem is that CRs often lack all the information required. Automatic prediction of CR fields can \ntherefore reduce the crash resolution process time. In this thesis, we use CR headings and \ndescriptions to predict the type of log files that should be attached to a CR. Our approach is to use \nmultilabel learning algorithms to train a machine learning model using a dataset from Ericsson’s \nCR database to predict the type of log files based on CR headings and descriptions. We use three \ndifferent pre-trained language models Bert, Telecom Bert, and Word2Vector to extract feature \nvectors from CR headings and descriptions and then feed these vectors to three different multilabel \nlearning algorithms, namely Binary Relevance (BR), Classifier Chain (CC), and Neural Network \n(NN). Then, we compare the performance of different feature sets. We found that the use of \nheadings alone with pre-trained language models Bert and Telecom Bert results in the best average \nAUC (0.70). The use of descriptions and headings and descriptions together as features resulted in \nan average AUC varying from 0.65 to 0.70. In general, the algorithms showed no significant \ndifference in their performances, but the choice of features impacts the performance. Also, the \nperformance of predicting each type of log is influenced by the use of keywords in headings and \ndescriptions that describe these files. We found that log types with a clear definition such as Key \nPerformance Indicators (KPI) Logs, Post-mortem Dumps (PMD), and execution traces can be \npredicted with higher accuracy.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.006 | 0.019 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.002 | 0.002 |
| Science and technology studies | 0.001 | 0.000 |
| Scholarly communication | 0.001 | 0.003 |
| Open science | 0.002 | 0.001 |
| Research integrity | 0.001 | 0.003 |
| Insufficient payload (model declined to judge) | 0.002 | 0.002 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".