Defining the system we want to evaluate.
The first problem is not choosing a model. It is deciding what a reliable customer-support system should be able to do.
Stage 1 in brief
- Start by defining the system behavior that matters, not by choosing a model.
- TinyBench-50 makes those behaviors concrete in 50 synthetic scenarios.
- Its results are evidence about a fixed setup, not a leaderboard.
AI customer support is often evaluated through a narrow lens: did the assistant provide a plausible answer, how quickly did it respond, or how much did the interaction cost?
Those measures matter, but they leave out the parts that determine whether a system is safe and useful in practice. A production support agent must use the right account data, follow policy, choose tools carefully, recover from failures, preserve identity controls, and escalate at the right moment.
What SupportBench is trying to solve
SupportBench evaluates complete system behavior in a synthetic environment. It provides a repeatable way to study how an agent, its tools, policies, memory, and recovery behavior work together, rather than treating a single language model as the whole product.
That distinction matters. A system can produce a fluent answer and still refund the wrong charge, override an identity control, invent a policy, or fail to recover when a billing dependency times out.
Customer support is a useful proving ground because technology, people, and enterprise readiness meet in one workflow. An agent must be capable, understandable to the customer, and safe inside the organization that deploys it.
Stage 1: a concrete, inspectable suite
TinyBench-50 v0.1 is the first research suite. It contains 50 fully synthetic customer-support scenarios spanning billing, identity and permissions, policy conflicts, retrieval, memory, recovery, enterprise constraints, and customer experience.
Each scenario specifies the customer message, synthetic account state, expected outcome, permitted tools, required arguments, explanation concepts, and the exact failure mode it is intended to expose.
| Scenario component | Why it matters |
|---|---|
| Customer message and account state | Tests decisions in context rather than generic response quality. |
| Permitted tools and required arguments | Catches unsafe actions, including a refund to the wrong charge. |
| Failure-mode notes and tags | Makes coverage and limitations inspectable. |
How the evaluation works
An agent receives the scenario and a restricted tool contract. It proposes an action, a customer-facing response, and tool calls. SupportBench executes those calls in the mock environment, then scores resolution, policy, escalation, tool use, and explanation.
The score is evidence about a fixed setup, not a declaration that one model is better than another. Model version, prompt, tools, retries, and harness design all matter. That is why the artifacts and methodology are public from the beginning.
| Score dimension | Question it answers |
|---|---|
| Resolution | Did the agent choose the expected outcome? |
| Policy and tool use | Did it use permitted tools with the required arguments? |
| Escalation and explanation | Did it hand off safely and clearly explain the outcome? |
Read the Stage 1 artifacts
- TinyBench-50 scenario CSV
- Methodology and limitations
- TinyBench-50 v0.1 manifest
- Scenario contribution guide
Concrete takeaways from Stage 1
- A useful support evaluation needs a concrete environment, not just a prompt and a preferred answer.
- Tool arguments matter. Refunding the wrong charge is a failure even when the final response sounds helpful.
- Policy, identity controls, recovery, and escalation are first-class system behaviors, not edge cases.
- Public scenarios and a documented rubric make the benchmark inspectable, but not automatically valid.
- Deterministic checks create a reproducible starting point; human calibration is required before interpreting explanation scores broadly.
What comes next
Stage 2 publishes the first local baseline. It shows why an evaluator needs to distinguish a correct safety escalation from an unnecessary handoff. Explanation coverage remains a separate, uncalibrated diagnostic. The next run will hold the scenarios and evaluator constant while comparing a second local agent.
If you build AI support systems, I would value your feedback on which failures, policies, and recovery cases deserve to be represented. Reach me at hey@deepnair.ai.