S poizvedovanjem obogateno generiranje besedil z domensko specifičnim doučevanjem velikih jezikovnih modeloveli

Developing an automated question-answering system to streamline customer support email handling presents an effective solution to reduce manual response times and effort. Currently, support agents respond to emails manually, which is time-consuming and labor-intensive. We tested several configurations to build a system capable of automatically answering these emails. The data used was provided by Zebra BI and consisted of a structured corpus of support emails and an unstructured corpus derived from product documentation. We compared two main approaches for tackling such problems. The first one fine-tuned a Large Language Model to answer emails directly. This approach contained a generator component only. The second one utilized the Retrieval-Augmented Generation (RAG) architecture which contained both the retriever and generator. The retriever, retrieved similar emails and append them to the context of an off-the-shelf LLM tasked with answering the question given the context from the previous similar emails. For the latter, we implemented a custom dual-decoder retriever model using the LoRA training technique and quantization. The dual-decoder retriever model generated embeddings for both the email and email answer passages using separate decoders and ranked them based on the cosine similarity. We developed six different question-answering system configurations. Some configurations utilized both the retriever and generator component, while others had only the generator. The best-performing configuration featured our custom dual-decoder retriever model, which improved the system's ability to retrieve relevant information from the domain-specific email corpus. This also showed that training a retriever model and utilizing the RAG architecture is more effective in comparison to fine-tuning an LLM, in cases where there is lower amount of data which is of lower quality. The most optimal retriever, built on the Llama-2-7B architecture using LoRA and 4-bit quantization, achieved a 0.53 Accuracy@100 and 0.032 MRR@100. In comparison, the state-of-the-art BGE-large-en retriever model scored 0.282 Accuracy@100 and 0.009 MRR@100 on the same domain. When paired with GPT-4o as the generator, the dual-decoder retriever received a 1282 ELO rating in our manual evaluation, while the BGE-large-en retriever scored 1256 with the same generator, and the fine-tuned LLM model scored 1071. The experiments showed that the dual-decoder retriever configuration provided the most accurate and contextually relevant answers, outperforming the state-of-the-art configuration. It also showed that decoder LLM models can be utilized for building a retriever model with relatively small amount of data. The final system, integrated into a Chrome extension, had a significant impact on Zebra BI's support workflows by automating a large portion of the response process. This not only reduced response times but also improved the accuracy and consistency of answers provided to customers. The extension is now used semi-automatically by Zebra BI's support team, freeing up agents to focus their attention on more complex issues.

Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.