The Calibration Gap: Model-Specific Confidence Thresholds for Reliable Customer Service LLMs
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
Every automated reply in customer service is a bet on a model's self-belief. When that confidence is wrong, brands either frustrate customers with bad answers or swamp agents with avoidable escalations. Deploying Large Language Models (LLMs) for customer service therefore hinges on calibrated confidence-i.e., the alignment between a model's stated confidence and the likelihood it is correct-because that signal governs automate-vs-escalate decisions. We benchmark three LLMs (GPT-4o, Claude-3.5-Sonnet, LLaMA-70B) on 2500 realistic hospitality cases spanning billing disputes, room service requests, experience complaints, and policy inquiries. Using an LLM-as-Judge framework that scores factual accuracy, process correctness, and completeness, we quantify calibration with reliability curves and confidence-accuracy gaps, and we sweep decision thresholds under a utility objective (accuracy minus escalation cost). Results show clear, category-dependent calibration differences: Claude-3.5-Sonnet is best aligned with the reliability diagonal overall; LLaMA-70B is most overconfident-especially for subjective experience complaints (gap 0.183)-and GPT-4o sits between these extremes. Threshold analysis indicates that optimal confidence cutoffs are model-specific and lower than common industry defaults: 0.55 for Claude-3.5-Sonnet, 0.60 for GPT4o, and 0.55-0.65 for LLaMA-70B (sensitive to escalation costs). These findings argue that confidence-based automation must consider calibration quality alongside accuracy, with model- and category-specific thresholds (stricter for subjective complaints) to balance automation gains against service reliability.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.002 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.002 |
| Science and technology studies | 0.002 | 0.000 |
| Scholarly communication | 0.001 | 0.002 |
| Open science | 0.002 | 0.001 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it