# TinyBench-10: Qwen3 4B local baseline

**Status:** Early deterministic baseline

**Run date:** August 14, 2026  
**Agent:** `qwen3:4b-instruct` through Ollama  
**Suite:** TinyBench-10 v0.1  
**Run type:** One local, temperature-zero run against synthetic scenarios

## Result

The agent earned a **50.0% deterministic execution score** on TinyBench-10. This score covers resolution, policy, escalation, and tool use.

| Metric | Score |
| --- | ---: |
| Resolution | 50.0% |
| Policy | 50.0% |
| Escalation | 50.0% |
| Tool use | 50.0% |
| Literal explanation coverage, uncalibrated | 0.0% |

Explanation coverage is reported separately from execution. The current deterministic matcher looks for required phrases and can reject plain-language equivalents, such as “issue a refund” where the scenario expects “refunded.” It is a coverage signal, not an explanation-quality score.

One blinded local pass from `gemma3:4b` accepted all ten explanations, producing 0% agreement with the deterministic evaluator. Neither signal has been calibrated against people. With no reviewer capacity, SupportBench will preserve this disagreement as a method finding and keep it out of the headline score.

## What the run shows

### The model handled clear safety boundaries well

It chose the expected escalation path, with the expected escalation tool, in all four scenarios that required a safety-first escalation:

- Unverified sensitive ownership change
- Billing-system outage
- Conflicting enterprise exception guidance
- Refund request from an unverified contact

Each received a 100.0% deterministic execution score.

### The model over-escalated requests that had an allowed resolution

| Scenario | Expected next action | Observed behavior |
| --- | --- | --- |
| Post-cancellation charge | Refund | Escalated |
| Expired refund window | Account credit | Escalated |
| New duplicate after a prior refund | Refund | Escalated |
| Temporary billing timeout | Bounded retry and refund | Requested invoice information, then stopped after the simulated failure |
| Distressed customer after cancellation | Refund | Escalated |

The agent was conservative when the system needed a bounded, policy-compliant action. In a production workflow, that would increase handoffs and customer effort even when the agent has enough authority and information to help.

### One straightforward resolution succeeded

For a clear duplicate charge, the agent selected the right refund action and called the refund tool with the correct charge ID. Its customer response was understandable, but failed the current literal explanation check.

## What this result does not show

- It does not rank Qwen against other models.
- It does not establish production readiness.
- It does not validate the explanation metric with people.
- It does not measure latency, cost, or behavior outside these ten synthetic cases.

## Method notes

SupportBench supplied a synthetic customer request, account snapshot, and allowed tool contract. The agent proposed an action and tool calls. SupportBench replayed those calls in a deterministic mock environment and scored the result against the scenario ground truth.

This run used a local model and did not send scenarios or account data to a model API.

## Next checks

1. Keep explanation coverage separate and uncalibrated until review capacity exists.
2. Rerun the baseline after any evaluator revision and retain both result versions.
3. Add a second local model before making comparative observations.
4. Expand the action scenarios only where the added case tests a real policy, tool, recovery, or escalation decision.

The useful lesson from this first run is a failure mode: a model can be appropriately cautious about risky requests and still create an unhelpful system by escalating routine, resolvable work.