The Calibration Gap: Model-Specific Confidence Thresholds for Reliable Customer Service LLMs
Bibliographic record
Abstract
Every automated reply in customer service is a bet on a model's self-belief. When that confidence is wrong, brands either frustrate customers with bad answers or swamp agents with avoidable escalations. Deploying Large Language Models (LLMs) for customer service therefore hinges on calibrated confidence-i.e., the alignment between a model's stated confidence and the likelihood it is correct-because that signal governs automate-vs-escalate decisions. We benchmark three LLMs (GPT-4o, Claude-3.5-Sonnet, LLaMA-70B) on 2500 realistic hospitality cases spanning billing disputes, room service requests, experience complaints, and policy inquiries. Using an LLM-as-Judge framework that scores factual accuracy, process correctness, and completeness, we quantify calibration with reliability curves and confidence-accuracy gaps, and we sweep decision thresholds under a utility objective (accuracy minus escalation cost). Results show clear, category-dependent calibration differences: Claude-3.5-Sonnet is best aligned with the reliability diagonal overall; LLaMA-70B is most overconfident-especially for subjective experience complaints (gap 0.183)-and GPT-4o sits between these extremes. Threshold analysis indicates that optimal confidence cutoffs are model-specific and lower than common industry defaults: 0.55 for Claude-3.5-Sonnet, 0.60 for GPT4o, and 0.55-0.65 for LLaMA-70B (sensitive to escalation costs). These findings argue that confidence-based automation must consider calibration quality alongside accuracy, with model- and category-specific thresholds (stricter for subjective complaints) to balance automation gains against service reliability.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.040 | 0.272 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.002 | 0.001 |
| Bibliometrics | 0.003 | 0.002 |
| Science and technology studies | 0.001 | 0.003 |
| Scholarly communication | 0.006 | 0.008 |
| Open science | 0.003 | 0.006 |
| Research integrity | 0.003 | 0.005 |
| Insufficient payload (model declined to judge) | 0.003 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".