The problem
The field has two dominant coding eval paradigms: single-shot benchmarks (HumanEval, MBPP, our own Coding Eval) and large agentic benchmarks (SWE-bench). Both leave a gap.
"Can the model write a correct fix in one try?" is a different question from "Can the model figure out what's broken, try something, learn from the failure, and keep going?"
Single-shot evals are saturated - frontier models score 90%+ and the signal is gone. SWE-bench measures the right behavior but at the wrong scale: 2,294 tasks, Docker containers, hours of compute. It's hard to run, hard to iterate on, and hard to explain.
| Benchmark | What it measures | The gap |
|---|---|---|
| HumanEval / MBPP | One-shot code correctness | Saturated. Doesn't measure what happens after a wrong guess. |
| SWE-bench | Multi-turn agentic repair on real repos | 2,294 tasks, Docker, hours of runtime. Hard to iterate and explain. |
| Agent-Gate | Multi-turn tool-use loop at small scale | Fills the middle: agentic behavior, no infrastructure, runs in 10 min. |
10
Tasks
154s
Total runtime
$0.29
Total API cost
How it works
Agent-Gate is a two-tier cascade system. Every task starts with Haiku (fast, cheap). If Haiku gets stuck, the harness escalates to Sonnet (capable, expensive) with a compact handoff. This mirrors the tiered routing pattern used in production AI products to balance cost and capability.
Task loaded into isolated workspace
Starter code + failing tests copied into a temp directory via Python's tempfile. No Docker required.
Haiku runs up to 4 turns
Haiku 4.5 gets the task description and 3 tools. Temperature 0.0. Tool output truncated to 50 lines.
Cascade check after each turn
Loop detected? Same failure 3× in a row? Explicit give_up? Turn limit hit? Any trigger fires escalation.
Compact handoff to Sonnet
Sonnet gets a ~200-token structured summary - not 2,000 tokens of raw Haiku history. Files touched + last test output.
Sonnet runs up to 6 more turns
Explicitly told: don't repeat what Haiku tried. Fresh approach, same 3 tools.
The three tools
| Tool | What it does | Restriction |
|---|---|---|
| read_file | Read any file in the workspace - code or tests | None - model can read freely |
| write_file | Overwrite a file with new content | Restricted to starter/ only - cannot touch tests |
| run_tests | Run pytest, return stdout + exit code | Exit code 0 = task solved |
Cascade trigger conditions
| Trigger | Condition | Why |
|---|---|---|
| Loop detection | Same tool + same args called twice in a row | Model is spinning - force escalation |
| Repeated failures | Identical test output 3 turns straight | No progress - escalation is correct |
| Explicit give_up | Model calls give_up() tool | Honest signal - rare but respected |
| Turn limit | Haiku used all 4 turns | Budget exhausted - Sonnet gets remaining turns |
Task design
Tasks are the heart of any eval. The 5 categories were designed to test specific failure modes - not just "can the model fix bugs" but "where does model reasoning break down?" Two tasks per category, 10 total. The model is told only that the tests are failing.
The agent is never told what the bug is. It must explore, diagnose, fix, and verify - the same loop Claude Code runs in practice.
| Category | Difficulty | What it tests | The trap |
|---|---|---|---|
| Single Bug - Obvious | Easy | Basic tool-use loop execution | Error message points directly at the bug - sanity check |
| Single Bug - Hidden | Medium | Exploration before acting | Bug is a logic error; error message is misleading |
| Two Bugs - Sequential | Medium | Persistence after partial success | Bug B only surfaces after Bug A is fixed |
| Test Understanding | Hard | Reading tests as requirements, not gates | Code is correct by one reading; test expects different behavior |
| Red Herring | Hard | Recovery from a wrong first diagnosis | Obvious-looking issue isn't the real bug - fixing it changes nothing |
Key findings
Real run: July 5, 2026. 154 seconds. $0.2946 total. These findings reflect actual model behavior - not estimates.
10/10
Tasks solved
90%
Cascade rate
40.8%
Cost saved vs Sonnet-only
| Finding | Detail |
|---|---|
| 10/10 solved - cascade worked perfectly | Every task was ultimately solved. Sonnet rescued all 9 tasks where Haiku hit its turn limit. The cascade trigger fired correctly every time. |
| Haiku hit its turn limit on 9/10 tasks | Only task_01 (single bug, obvious) was solved by Haiku alone. This suggests the 4-turn budget is tight for most tasks, or Haiku needs a more directive initial exploration prompt. |
| 40.8% cost savings vs Sonnet-only | $0.29 actual vs $0.50 if everything ran on Sonnet from the start. The cascade saves money even with a 90% escalation rate - because Haiku's cheap exploration turns are far less expensive than Sonnet's. |
| Self-correction rate: 0.0 | The model committed to its first fix per phase rather than iterating in-phase. The cascade itself is doing the correction work - escalation replaces in-phase retry. Worth investigating in v3. |
| Sonnet rescue rate: 100% | Every task that escalated to Sonnet was solved. The compact handoff summary was sufficient context - Sonnet didn't need raw Haiku history to take a different approach. |
The cascade saves 40.8% vs Sonnet-only - even with a 90% escalation rate. The math works because Haiku turns are ~6× cheaper than Sonnet turns.
Design decisions
Every decision involved a real tradeoff. These are the six most important ones - what was chosen, why, and what was left on the table.
| Decision | Choice | Why | Tradeoff |
|---|---|---|---|
| No Docker | Python tempfile + subprocess | Sufficient isolation for single-file tasks. Runs anywhere Python and pytest are installed. | Doesn't scale to multi-file codebases that import across directories. |
| Pre-computed results | Static publicSnapshot.ts | Live API calls on every page view would cost money and require server infrastructure. Pre-computed is instant and free to serve. | Results go stale as models update. Noted in methodology. |
| Compact cascade handoff | ~200-token summary to Sonnet | Sending raw Haiku history (2,000+ tokens) wastes money and risks Sonnet repeating Haiku's mistakes. | Sonnet loses visibility into Haiku's exact reasoning. In practice the summary is sufficient. |
| Temperature 0.0 | Fully deterministic | Agentic tool-use decisions need determinism for reproducible results. | Reduces model creativity - acceptable for tasks with clear correct answers. |
| Single run per task | No majority vote | 3× majority vote would triple cost ($0.90 vs $0.29). Variance mitigated by temperature 0.0. | Some run-to-run variance remains. Would address in v3. |
| 50-line tool output cap | Truncate pytest output | Model only needs the bottom of pytest output - that's where the error is. Saves tokens every turn. | Model can't see full test history. Last 50 lines always contain the relevant failure. |
PM signal
Agent-Gate was built to demonstrate PM fluency across the full stack of AI product work - not just writing prompts, but designing evals, understanding cost tradeoffs, and thinking about agentic failure modes at depth.
| JD requirement | How Agent-Gate addresses it |
|---|---|
| Personally built agentic evals | Built from scratch: real tool-use loop, real cascade logic, real pytest grading. Not a framework wrapper. |
| SWE-bench-style task suites | Same concept - broken codebase, autonomous agent, test-driven grading - at a scale that's reproducible and explainable. |
| Model performance on coding tasks | Direct comparison: Haiku vs cascade vs Sonnet-only. Per-task traces show exactly where each model succeeded or failed. |
| Cost and latency tradeoffs | Cascade efficiency metric quantifies savings from tiered routing. Every turn logged with exact tokens, cost, and latency. |
| Eval design taste | 5 categories that test failure modes no existing benchmark isolates: loops, red herrings, sequential bugs, test misalignment. |
| PM + technical communication | Same page serves a general visitor (cost comparison, headline numbers) and a technical reviewer (step-by-step trace replay with full token accounting). |
Building and running this eval end-to-end cost less than a cup of coffee. That's the point - high-signal evals don't require massive infrastructure.
What's next
The current version proves the core concept. v3 would scale it toward something closer to production-grade agentic evaluation.
| Feature | Why it matters |
|---|---|
| Multi-file codebases | Real bugs span files - the agent needs to navigate a directory tree, not just one module. |
| TypeScript tasks | Tests whether the cascade generalizes across languages beyond Python. |
| Web search tool | Closer to real Claude Code usage - model looks up docs and error messages mid-task. |
| Live streaming demo | User picks a task, watches the agent run in real time. Static results become interactive. |
| Tighter Haiku prompt | Investigate whether a more directive initial prompt (read tests first, form hypothesis before writing) improves Haiku's close rate. |
| 3× majority vote | Run each task 3 times, take majority. Reduces remaining variance from single-run results. |