hubagentic eval · researchdescriptionView Product Doc

Agent-Gate: Agentic Eval

Can Claude autonomously fix broken code without being told what's wrong? A 10-task harness measuring multi-turn tool use, cascade routing, and cost efficiency.

check_circle10/10tasks solved

cached6.8avg turns

swap_horiz90%cascade rate

savings41%saved vs Sonnet-only

Results from latest local run

Why this benchmark exists

Single-shot evals miss the loop

HumanEval asks: can the model write correct code once? Agent-Gate asks: can it figure out what's wrong, fix it, and keep going?

SWE-bench is too heavy to run

2,294 tasks, Docker, hours of compute. Agent-Gate runs end-to-end in under 10 minutes with no infrastructure.

Cascade efficiency is unmeasured

No benchmark directly measures whether routing easy tasks to cheap models actually pays off. This one does.

Does the cascade pay for itself?

The cascade starts with cheap Haiku and escalates to Sonnet only when stuck, capturing most of Sonnet's capability at a fraction of the price.

Haiku Only

$0.133

~7/10 solved

Cheap but gets stuck on hard tasks

starBest value

Cascade (Actual)

$0.295

10/10 solved

Best balance of cost and capability

Sonnet Only

$0.498

~8/10 solved

Most capable but 60%+ more expensive

trending_down

Cascade saved 41% vs Sonnet-only

upgrade

Cascade rescued 100% of tasks Haiku failed

Eval at a glance

check_circle

100%

Pass Rate

10 of 10 tasks

cached

6.8

Avg Turns

per solved task

swap_horiz

9/10

Cascade Triggered

tasks escalated to Sonnet

volunteer_activism

100%

Rescue Rate

Sonnet solved after Haiku failed

undo

0.0

Self-Corrections

avg per task

timer

2219ms

Turn Latency

avg across all turns

bolt

10/10

First-Attempt Pass

first write passed tests

payments

$0.295

Total Cost

all 10 tasks, cascade

All 10 tasks

Click any row to replay the full step-by-step agent trace.

	Task	Difficulty	Turns	Cascade	Cost

single bug obvious Single Bug — Obvious	Easy	4(4H)	Haiku only	$0.0066	expand_more
single bug obvious Single Bug — Obvious	Easy	8(4H+4S)	swap_horizHaiku turn limit	$0.036	expand_more
single bug hidden Single Bug — Hidden	Medium	7(4H+3S)	swap_horizHaiku turn limit	$0.023	expand_more
single bug hidden Single Bug — Hidden	Medium	7(4H+3S)	swap_horizHaiku turn limit	$0.025	expand_more
two bugs sequential Two Bugs — Sequential	Medium	8(4H+4S)	swap_horizHaiku turn limit	$0.039	expand_more
two bugs sequential Two Bugs — Sequential	Medium	7(4H+3S)	swap_horizHaiku turn limit	$0.040	expand_more
test understanding Test Understanding	Hard	7(4H+3S)	swap_horizHaiku turn limit	$0.027	expand_more
test understanding Test Understanding	Hard	8(4H+4S)	swap_horizHaiku turn limit	$0.052	expand_more
red herring Red Herring	Hard	6(4H+2S)	swap_horizHaiku turn limit	$0.023	expand_more
red herring Red Herring	Hard	6(4H+2S)	swap_horizHaiku turn limit	$0.022	expand_more

When and why did Haiku escalate?

Trigger breakdown (9 cascades)

Haiku turn limit9

Most common: repeated failures - Haiku applied the same wrong fix until the cascade fired.

Sonnet rescue rate

9/9

cascades resolved

Sonnet received a compact 200-token handoff - not raw Haiku history - so it approached each task fresh instead of repeating failed attempts.

How this eval works

task

Task design

→10 tasks across 5 categories
→Obvious bugs, hidden errors, sequential bugs, test traps, red herrings
→Each task: isolated Python workspace + failing test suite
→Agent is never told what the bug is

build

Tool loop

→3 tools: read_file, write_file, run_tests
→Agent calls tools iteratively each turn
→Stops when tests pass, turn limit hit, or give_up called
→No hints given between turns

swap_horiz

Cascade logic

→Haiku runs first - up to 4 turns
→Escalates on: loop detection, 3 identical failures, or give_up
→Sonnet gets a compact ~200-token handoff summary
→Sonnet runs up to 6 additional turns

payments

Cost calculation

→Haiku: $0.80/M input · $4/M output
→Sonnet: $3/M input · $15/M output
→Counterfactual = all turns priced at one model's rate
→Savings = (Sonnet-only − actual) / Sonnet-only

Reproducibility

Haiku: claude-haiku-4-5-20251001Sonnet: claude-sonnet-4-6Temperature: 0Haiku turns: 4Sonnet turns: 6Output cap: 50 lines

To rerun: ANTHROPIC_API_KEY=your_key python src/eval-engine/run_agentic_eval.py

What's next

folder_open

Multi-file codebases

Real bugs span files. The agent needs to navigate a directory tree, not just one module.

javascript

TypeScript tasks

Tests whether the cascade generalizes across languages beyond Python.

Web search tool

Closer to real Claude Code usage - model looks up docs and error messages mid-task.

live_tv

Live streaming demo

User picks a task, watches the agent run in real time via streaming.

device_hub

Third-tier routing

Add a third model tier for cross-provider cascade comparison.

groups

3× majority vote

Run each task 3 times, take majority. Reduces variance from single-run results.

What it cost to build this

Every build session tracked - model, token count, API spend.

Session	Goal	Tokens	Cost
S1	Full planning	~65k	$0.29
S2	10 task definitions	~52k	$0.23
S3	Python harness	TBD	TBD
S4	TypeScript types + snapshot	TBD	TBD
S5	TraceReplayViewer	TBD	TBD
S6	AgenticEvalShowcase + page	TBD	TBD
S7	Run harness + deploy	TBD	TBD

coffee

Building and running this eval cost less than a cup of coffee.

Total: ~117k tokens tracked (Sessions 1-2). Full cost including eval runs: ~$0.50-$0.90 est.