Evaluating the Quality of Large Language Model-Generated Explanations in Recommendation Tasks: A Multi-Dimensional Comparative Analysis
Bibliographic record
Abstract
The integration of large language models (LLMs) into recommendation systems has introduced new possibilities for generating natural language explanations that accompany recommended items. While prior research has explored methods of leveraging LLMs for explanation generation, limited attention has been given to systematically evaluating the quality of these explanations across different models, domains, and prompting strategies. This paper presents a multi-dimensional comparative analysis of LLM-generated recommendation explanations, examining four commercially available and open-source LLMs across three public recommendation datasets. A structured evaluation framework is proposed that encompasses four quality dimensions: faithfulness, informativeness, persuasiveness, and personalization. The evaluation employs both automatic metrics (BLEU-4, ROUGE-L, BERTScore) and human annotation protocols involving 12 trained evaluators. The experimental results indicate that larger-parameter models produce more informative and faithful explanations, though the gap narrows substantially when context-enhanced prompting strategies are applied. Automatic metrics show moderate correlation with human judgments on informativeness and faithfulness but limited alignment on persuasiveness and personalization. These findings offer practical guidance for selecting appropriate LLMs and prompting strategies in explanation-augmented recommendation applications.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.008 | 0.002 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.000 |
| Bibliometrics | 0.000 | 0.006 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.001 |
| Open science | 0.001 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".