Evaluating AI in Customer Support: Statistical Analysis

Can a bot really make customer support more efficient or is it just hype?

AI generated via Sora

Over the last few months, I’ve been digging into this question through a side project involving simulated support case data and AI-assisted reviews. The goal? To evaluate whether introducing an AI bot for ticket pre-processing could tangibly improve operational metrics like Average Handle Time (AHT) and customer satisfaction.

This wasn’t tied to any company or product. Just me, some datasets, a few models, and a growing obsession with causal inference.

Problem Setup

If you’ve ever contacted customer support for help with your online account, you’ve probably triggered one of these common support scenarios whether you knew it or not.

Imagine a support system that handles three broad types of incoming cases:

1. User Activity Checks — Think: “Did someone just try to log in from Tokyo while you were asleep in New York?” The system needs to validate that kind of unusual activity.
2. Account Sharing Suspicion — Say you’ve logged in on a bunch of devices in different cities. The system might flag your account for potentially being shared, even if it’s just you switching between work, home, and mobile.
3. Complex Verification Cases — These are more serious. Maybe someone tried to reset your password using suspicious techniques or mimicked your identity. These typically require manual review by human agents because they often involve social engineering.

Enter “SupportIQ” (totally made that up but if such system exists lmk!), an AI system that pre-reviews incoming tickets and categorizes them. In some cases, it resolves them and in others, it tags them for manual follow-up.

I created a synthetic dataset (10k rows) simulating bot-reviewed vs manually reviewed support tickets and their downstream metrics like:

• AHT (Average Handle Time in seconds)
• CSAT (Customer Satisfaction Score, 0-100 scale)
• RES (Resolution Confidence Score, 0-100 scale)
• Ticket Type
• Bot vs Manual Review (binary)

Analytical Approach

In this analysis I only looked through one metric Average Handle Time aka AHT.

Get Hitali Shah’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

To cut through noise and anecdotal observations, I followed a layered statistical workflow:

1. Exploratory Data Analysis
2. Permutation Testing for Robust Comparison
3. Bayesian Causal Inference (Using CausalImpact)
4. Confidence Intervals + Posterior Probabilities

1. Exploratory Insights

Here’s what the synthetic data looked like on the surface:

• Bot-reviewed cases showed a minor drop in AHT on average (~2 seconds).
• In specific case types (User Activity & Account Sharing), bots saved up to 90 seconds.
• Satisfaction scores were modestly higher in bot-handled scenarios.

Figure 1: Mean AHT by Case Type and Review Method

2. Permutation Testing: Is the Difference Real?

Permutation tests are ideal when dealing with imbalanced groups, which was the case here, about 65% of the data were manually reviewed.

Hypothesis:

• H₀: There’s no difference in AHT between bot-reviewed and manual cases.
• H₁: Bot-reviewed cases reduce AHT.

Test Setup:

• Ran 10,000 permutations per case type.
• Measured observed differences vs null distribution.
• Used one-sided testing since we’re only interested in improvements.

Bot review significantly reduced handle times in simpler case types. In complex cases, it slightly increased time likely due to post-bot follow-up steps.

3. Bayesian Causal Inference (Using CausalImpact)

To move beyond just comparing means, I turned to causal inference to determine whether the observed improvements from SupportIQ were actually caused by the bot or just random fluctuations in the data.

For this, I used the CausalImpact library in Python, a port of Google’s original R package. It’s built for answering a very specific and powerful question:

“What would have happened if the intervention (in this case, the AI bot) had not occurred?”

How It Works:

CausalImpact models your time series as two phases:

• Pre-Intervention: A baseline period where the bot wasn’t active.
• Post-Intervention: The time after bot deployment.

It then builds a Bayesian structural time series model using the pre-period to predict what the post-period would have looked like without the bot. The difference between actual and predicted outcomes becomes the estimated causal effect.

Input Data:

• Response variable: Average Handle Time (AHT), aggregated weekly.
• Covariates: Case volume, issue type ratios, satisfaction scores.
• Intervention point: The simulated “go-live” date of the bot.

Results:

The model showed a statistically significant drop in AHT post-intervention, with high confidence.

The bot likely caused a ~2-minute reduction in handle time for specific case types, with a very high probability that the observed drop wasn’t due to chance.

What Did I Learn?

• Statistical rigor beats gut feeling: Some gains were not obvious until I ran a permutation test and causal model.
• False negatives are real: High Type II errors in basic stats can mask genuine improvements causal modeling helps here.

Drop your thoughts in the comments! 👇