Confidence Thresholds That Don't Flood Your Human Queue

The real problem isn't a bad answer — it's the wrong escalation rate

When a support team first deploys an AI agent, they usually set one confidence threshold — say, 70% — and move on. Two weeks later they're looking at two failure modes simultaneously: the AI is confidently answering billing edge-cases it shouldn't touch, and it's escalating routine password-reset questions that it could handle fine. Both failures come from treating confidence threshold as a single global knob.

This post is about calibrating that knob — actually, replacing it with several — so your human queue gets only the escalations that genuinely need a person.

Understand what the confidence score actually measures

Chattering's confidence score reflects how closely the retrieved source content matches the question and how unambiguous the generated answer is given that content. A score of 0.82 on a straightforward refund policy question means something different from 0.82 on a question about an enterprise contract clause that shows up twice in your docs with contradictory language. Same number, very different risk.

Before you set any threshold, know what low confidence actually signals in your product. Pull a sample of 50 recent low-confidence responses and read them. You'll usually find three clusters: ambiguous questions the AI understood but couldn't source, questions outside your documented scope, and questions where the docs themselves are inconsistent. Each cluster deserves a different response — not a single threshold.

Segment thresholds by intent category, not globally

Map your ticket intents into consequence tiers before you touch any configuration.

High consequence intents — billing disputes, cancellation, data deletion, security — should escalate at a higher threshold. If the AI is only 75% confident on a cancellation question, a human should see it. The cost of a wrong answer here is churn or legal exposure.

Medium consequence intents — feature how-tos, integration setup, account settings — can tolerate a lower threshold. A slightly imperfect answer on "how do I add a team member" usually gets corrected by the user in a follow-up.

Low consequence intents — status page questions, hours of operation, pricing lookup — can go even lower, or you can skip AI-confidence escalation entirely and only escalate on explicit user requests.

In Chattering, you configure this by tagging intents and setting threshold ranges per tag. Start with three tiers. You can refine to more once you have enough volume to see patterns.

Set your baseline by reading the failure, not the rate

A lot of teams set thresholds by aiming for an escalation rate — "we want AI to handle 60% of tickets" — and work backwards. That's the wrong direction. An escalation rate is a lagging indicator of threshold correctness; it doesn't tell you whether the right tickets are escalating.

Instead, start with a two-week observation window. Set your initial thresholds conservatively (escalate more), then review what actually arrived in the human queue. For each escalated ticket, mark it: did this genuinely need a human, or could the AI have answered it? After two weeks you'll have a ground-truth signal. Move the threshold down only on the intents where agents kept marking tickets as unnecessarily escalated.

This is slower than guessing, but it means you're calibrating on your customers and your docs, not on a benchmark that has nothing to do with your team.

Add a secondary escalation trigger: user sentiment

Confidence score alone misses cases where the AI answered correctly but the customer is still frustrated. A user who sends three messages in under two minutes, or who types "this is ridiculous" after a correct answer, should escalate regardless of confidence score.

Chattering supports escalation rules based on conversation signals — repeated follow-ups, explicit frustration language, or the customer explicitly asking for a human. Layer these on top of confidence thresholds so your queue captures the emotional escalations that pure confidence scoring can't see.

Run a weekly threshold review, not a quarterly one

Your docs change. Your product changes. A threshold that was correctly calibrated in March will drift by May if you shipped a new billing model and updated three help articles. Confidence scores will shift because the underlying source content shifted.

• Weekly: Check the escalation reason breakdown in Chattering's inbox view. If one intent category is spiking, find out why before assuming the threshold is wrong — sometimes it's a doc gap.
• Monthly: Pull CSAT on AI-resolved tickets vs. escalated-then-resolved tickets. If AI-resolved CSAT is dropping, your threshold is probably too permissive.
• On every major product change: Manually test 10–15 questions in the affected area and read the confidence scores before those changes reach customers.

The threshold that protects both sides

The goal isn't maximum AI containment. It's matching ticket complexity to the right responder. A high-consequence ticket handled poorly by AI costs you more than an agent handling it directly. A routine ticket escalated unnecessarily costs your agent time and slows response for tickets that actually need them.

Get the segmentation right first. Then calibrate the numbers. Then keep checking. The threshold is a maintenance task, not a launch task.