Gloss: Guiding Large Language Models to Answer Questions from System Logs
Bibliographic record
Abstract
System logs contain valuable information and they have emerged as one of the most crucial data sources for system monitoring aimed at enhancing service quality. IT support teams and system administrators are in dire need of an intelligent log-based QA system to help them quickly identify, diagnose, and resolve issues. In this paper, we propose a novel method for constructing log-based question-answering (QA) data using large language models, addressing challenges associated with limited dataset size and diversity in existing log-based QA systems. Our pipeline consists of three steps: generating questions, answering log questions, and refining question-answer pairs. The purpose of the generating questions is to create a diverse set of log-related queries that cover a wide range of potential issues. The second step, answering log questions, aims to extract relevant information from the logs to address the generated questions. This step ensures accurate and context-aware responses. Refining question-answer pairs is intended to improve the overall quality and consistency of the generated log-based QA data. We present a case study using ChatGPT to generate a new dataset, LogQuAD, containing over 28,000 question-answer pairs derived from more than 31,000 raw logs, representing a significant increase compared to existing datasets like LogQA. In our experimental setting, we sample half of the data as the training set and use memory-effect fine-tuning to fine-tune the model, named Gloss. Experimental results show that our method can generate high-quality log-based QA data, leading to improved performance of log-based QA models. Notably, our fine-tuned 7B model outperforms the LLaMA-65B model. This approach can potentially save valuable time for IT support teams and system administrators, enabling proactive problem resolution and optimal system performance.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.000 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.001 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".