Write Docs That Teach Your AI When Rules Bend


TL;DR
When you add an AI agent, CSAT often drops for a week because the survey pool shifts, not because quality fell. Separate deflected tickets from escalated ones, segment scores by resolution path, and give the data at least two weeks before drawing conclusions.
You ship your AI agent on a Monday. By Friday, your CSAT is down three points and someone's already asking whether you should turn it off. Before you do anything, recognize what actually happened to the data you're reading.
Before the AI agent, every ticket went to a human. Every resolved ticket was eligible for a CSAT survey. Your baseline score reflected that entire mix — quick billing fixes, tricky integrations, the full range.
After the agent goes live, the tickets that reach humans are no longer a representative sample. The agent deflects the routine ones. What lands in the human queue is heavier: edge cases, billing disputes, frustrated users who already tried self-service and failed. You haven't made support worse. You've changed which tickets humans touch, which changes who gets a survey, which changes your aggregate score.
That's the core problem. Your CSAT denominator shifted without anyone updating the dashboard.
Survey pool composition. If your AI agent deflects 30% of tickets and closes them without escalation, the humans who remain are handling the harder 70%. Their CSAT will be lower than the old aggregate — not because they're performing worse, but because the easy wins are no longer in the pool.
Survey timing and trigger. Some teams only survey tickets closed by a human. Others survey any closed conversation, including AI-only ones. If you switched trigger logic when you deployed — even accidentally — you're comparing two different measurement setups.
Novelty friction. Some customers will rate an AI interaction lower simply because it surprised them, regardless of whether it answered correctly. This friction is real but temporary. It tends to compress within two to three weeks as users learn to expect the agent.
The fix is segmentation, not averaging.
In your shared inbox, tag every closed ticket by resolution path: AI-only resolved, AI-to-human escalated, human-only. Run CSAT separately for each segment. What you're looking for:
Chattering's shared inbox lets you filter conversations by resolution path and export per-segment CSAT without custom tooling. That segmentation should be in place before launch, not after the panic.
If the AI agent hands off to a human without context, the customer repeats themselves. That repetition is a CSAT killer — and it's entirely avoidable. When Chattering escalates a conversation, the full thread and the pages the agent cited transfer to the human agent automatically. The human doesn't start from zero.
An escalation that feels seamless to the customer can still score well. An escalation that makes them re-explain their issue twice will score poorly and deserve to.
If your escalated CSAT is dragging your overall number down, check the handoff first. Is the human seeing the full context before they respond? Are they acknowledging what the AI already tried?
One week is not enough data to act on after a major workflow change. Survey response rates are lumpy. Customer mix varies by day of week. Any single-week read will have variance you can't distinguish from signal.
Set a two-week minimum before you make any configuration changes based on CSAT. During that window, watch three things: AI-only resolution rate, escalation rate, and per-segment CSAT. If AI-only resolution CSAT is low and stable across week one and week two, the agent is failing at quality — fix its knowledge base or scope it back. If it's low in week one and recovering in week two, you're seeing novelty friction burn off.
Say your team handles 600 tickets a month pre-launch. The AI agent deflects 200 of those in its first full month. If your human-only CSAT holds steady and your AI-only CSAT is within a few points of it, you've reduced load without hurting quality. Your aggregate may still look flat or slightly lower — because the escalated segment is dragging it — but the underlying picture is positive.
The goal isn't to protect your aggregate CSAT score. It's to understand what's moving it and why. Segmented data makes that possible. Aggregate data just creates fire drills.
Before you present the post-launch CSAT to your team or your investors, attach the segmentation. The number without context is the problem, not the AI agent.
Not on a single week of data — first check whether the drop is in the AI-only segment, the escalated segment, or your human-only segment, because each has a different cause and fix.
Send the survey immediately on close, the same way you would for a human-resolved ticket, and make sure the customer knows the conversation was handled by an AI so their rating reflects the actual experience.
That's a real signal worth acting on — review which topics the agent is handling and either expand its knowledge base for those cases or narrow its scope to the queries it resolves reliably.