Evaluating AI-Native Customer Support Systems.
Customer support is becoming one of the first business functions to be fundamentally rebuilt around AI.
In brief
- AI support should be evaluated as a complete system, not only as a model.
- SupportBench focuses on tools, policy, memory, recovery, escalation, and trust.
- The work is early-stage, synthetic, and designed to be inspectable.
Every week we see better foundation models, new agent frameworks, and startups pushing the boundaries of what AI support agents can do. Yet one question remains surprisingly difficult to answer:
How do we know if an AI customer support system is actually good?
Today, teams often rely on metrics like resolution rate, CSAT, latency, or token cost. These are useful, but they don’t tell the whole story. A production AI support system also needs to:
- follow company policy consistently;
- use tools correctly;
- retrieve the right knowledge;
- remember relevant context across conversations;
- escalate when appropriate;
- recover gracefully from failures; and
- earn customer trust.
| Common metric | What it can miss |
|---|---|
| Resolution rate | Whether the action followed policy or affected the correct account record. |
| CSAT | Whether the system used tools safely and escalated when it should. |
| Latency or cost | Whether the response was grounded in the right knowledge and context. |
Unlike software engineering, coding, or general language models, AI-native customer support doesn’t yet have widely adopted open benchmarks, datasets, or evaluation standards.
That’s the gap I want to help close.
This is early-stage independent research, not a finished standard or a model leaderboard. The aim is to make the methods, evidence, and limitations clear as the work develops.
My mission
My long-term mission is simple:
Advance the science and engineering of AI-native customer support.
Over the next few days, I’ll be building in the open to explore what makes AI support systems reliable, trustworthy, and genuinely useful.
What I’m building
1. Benchmarks
I’m building open benchmarks for evaluating AI-native customer support systems using synthetic but realistic customer scenarios.
Rather than asking “Which model is best?”, I want to answer questions like:
- Which agent architecture performs best?
- Does memory actually improve outcomes?
- How should support agents be evaluated?
- What failure modes appear most often?
- When should an agent escalate to a human?
2. Research
I’ll publish experiments, technical write-ups, and benchmark results as I learn.
Whenever possible, conclusions will come from reproducible experiments using open datasets and tooling.
| Research question | Evidence I want to produce |
|---|---|
| Does memory help? | Scenario-level outcomes and failure patterns with and without memory. |
| When should an agent escalate? | Traceable examples of safe escalation and unsafe overreach. |
| Which architecture is more reliable? | Reproducible comparisons that hold the scenario and evaluator constant. |
Principles
A few principles will guide this work:
- Build in public.
- Use synthetic data and publicly available information.
- Make experiments reproducible.
- Optimize for practical engineering value over flashy demos.
- Share failures as openly as successes.
Continue to SupportBench, Stage 1: Defining the System We Want to Evaluate.