Evaluating AI-Native Customer Support Systems.
Customer support is becoming one of the first business functions to be fundamentally rebuilt around AI.
Every week we see better foundation models, new agent frameworks, and startups pushing the boundaries of what AI support agents can do. Yet one question remains surprisingly difficult to answer:
How do we know if an AI customer support system is actually good?
Today, teams often rely on metrics like resolution rate, CSAT, latency, or token cost. These are useful, but they don’t tell the whole story. A production AI support system also needs to:
- follow company policy consistently;
- use tools correctly;
- retrieve the right knowledge;
- remember relevant context across conversations;
- escalate when appropriate;
- recover gracefully from failures; and
- earn customer trust.
Unlike software engineering, coding, or general language models, AI-native customer support doesn’t yet have widely adopted open benchmarks, datasets, or evaluation standards.
That’s the gap I want to help close.
This is early-stage independent research, not a finished standard or a model leaderboard. The aim is to make the methods, evidence, and limitations clear as the work develops.
My mission
My long-term mission is simple:
Advance the science and engineering of AI-native customer support.
Over the next few days, I’ll be building in the open to explore what makes AI support systems reliable, trustworthy, and genuinely useful.
What I’m building
1. Benchmarks
I’m building open benchmarks for evaluating AI-native customer support systems using synthetic but realistic customer scenarios.
Rather than asking “Which model is best?”, I want to answer questions like:
- Which agent architecture performs best?
- Does memory actually improve outcomes?
- How should support agents be evaluated?
- What failure modes appear most often?
- When should an agent escalate to a human?
2. Research
I’ll publish experiments, technical write-ups, and benchmark results as I learn.
Whenever possible, conclusions will come from reproducible experiments using open datasets and tooling.
Principles
A few principles will guide this work:
- Build in public.
- Use synthetic data and publicly available information.
- Make experiments reproducible.
- Optimize for practical engineering value over flashy demos.
- Share failures as openly as successes.