An Industrial Study on Predicting Crash Report Log \nTypes Using Large Language Models
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
Software crashes and failures take a fair amount of effort and time to resolve. Software developers \nuse information submitted in crash reports (CRs) to conduct root cause analysis of faults. The \nproblem is that CRs often lack all the information required. Automatic prediction of CR fields can \ntherefore reduce the crash resolution process time. In this thesis, we use CR headings and \ndescriptions to predict the type of log files that should be attached to a CR. Our approach is to use \nmultilabel learning algorithms to train a machine learning model using a dataset from Ericsson’s \nCR database to predict the type of log files based on CR headings and descriptions. We use three \ndifferent pre-trained language models Bert, Telecom Bert, and Word2Vector to extract feature \nvectors from CR headings and descriptions and then feed these vectors to three different multilabel \nlearning algorithms, namely Binary Relevance (BR), Classifier Chain (CC), and Neural Network \n(NN). Then, we compare the performance of different feature sets. We found that the use of \nheadings alone with pre-trained language models Bert and Telecom Bert results in the best average \nAUC (0.70). The use of descriptions and headings and descriptions together as features resulted in \nan average AUC varying from 0.65 to 0.70. In general, the algorithms showed no significant \ndifference in their performances, but the choice of features impacts the performance. Also, the \nperformance of predicting each type of log is influenced by the use of keywords in headings and \ndescriptions that describe these files. We found that log types with a clear definition such as Key \nPerformance Indicators (KPI) Logs, Post-mortem Dumps (PMD), and execution traces can be \npredicted with higher accuracy.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.003 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.000 |
| Bibliometrics | 0.002 | 0.002 |
| Science and technology studies | 0.001 | 0.000 |
| Scholarly communication | 0.000 | 0.001 |
| Open science | 0.002 | 0.000 |
| Research integrity | 0.001 | 0.002 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it