# SupportBench methodology

## Purpose

SupportBench evaluates complete AI-native customer-support systems in a synthetic, deterministic environment. It is designed to study system behavior: the chosen outcome, tool use, policy adherence, recovery, escalation, and customer-facing explanation.

It is not a measure of general intelligence, real-world customer satisfaction, production safety, or an organization’s support quality.

## Suites

- **TinyBench-10** is the stable smoke-test suite. Its transparent baseline is used to verify the harness.
- **TinyBench-50 v0.1** is the research suite. It combines the 10 core cases with 40 expanded cases spanning billing, identity, permissions, retrieval, memory, recovery, enterprise constraints, and customer experience.

All companies, accounts, policies, and records are fictional and synthetic.

## Evaluation protocol

For every scenario, an agent receives a customer message, synthetic account snapshot, and a restricted tool contract. It proposes an action, response, and tool calls. SupportBench executes those calls in the synthetic environment, then scores five dimensions equally:

1. **Resolution**: does the agent choose the expected outcome?
2. **Policy**: does it use only permitted tools, respect declared recovery limits, and satisfy required tool arguments?
3. **Escalation**: does it escalate exactly when the scenario requires it?
4. **Tool use**: does the expected tool succeed with required arguments?
5. **Explanation**: does the customer-facing response cover the required concepts?

The overall score is the unweighted mean of those five dimensions. Category scores aggregate scenario scores by tags; they are diagnostic, not independent statistical estimates.

## Limitations

- The scenarios are hand-authored and initially small in number.
- Synthetic account state cannot reproduce the full ambiguity of production systems.
- The explanation check is deterministic phrase matching. It must be calibrated against human review before comparative claims are made.
- Results depend on model version, prompt, tool contract, harness, date, retries, and runtime configuration.

## Reporting standard

Any published result should include the suite version, commit SHA, agent and model identifiers, prompt or agent configuration, tool contract, runtime date, retry policy, scorecard, traces where practical, and known limitations. Results should describe the tested configuration, not make claims about a model or company in general.