Claude Code Coding Eval
A 25-task benchmark testing how Claude models handle real developer work: fixing bugs, adding features, refactoring, and following vague instructions.
Why this benchmark exists
HumanEval is saturated
Models score 90%+ - it can't discriminate between frontier models anymore. Also likely contaminated.
SWE-bench is complex
2,294 tasks across 12 Python repos. Useful but expensive to run and conflates retrieval skill with coding skill.
None test the loop
No major benchmark tests whether a model asks for clarification, handles ambiguity, or catches multi-step bugs.
Overall scores · 25 tasks
Three-model comparison
Last run: July 3, 2026
Claude Haiku 4.5↑ top
76%
19 / 25 tasks
Claude Sonnet 4.6
72%
18 / 25 tasks
By category
Where each model leads and lags
| Category | Haiku | Sonnet 4.6 |
|---|---|---|
Bug Fixing 5 tasks | 4/5 80% | 3/5 60% |
Feature Addition 5 tasks | 5/5 100% | 5/5 100% |
Refactor 5 tasks | 5/5 100% | 5/5 100% |
Ambiguous Instructions 5 tasks | 0/5 0% | 0/5 0% |
Multi-step Debug 5 tasks | 5/5 100% | 5/5 100% |
Key finding
Haiku 4.5 outscored Sonnet 4.6 on these tasks, 76% to 72%. Both models aced feature addition and refactoring. The sharpest finding was ambiguity handling: neither model asked a single clarifying question across all 5 ambiguous tasks. Both picked an interpretation and ran with it, every time.
Failure analysis
How models fail, not just that they fail
Wrong fix
Code was changed but the bug or requirement is still not met. The model diagnosed the symptom, not the cause.
Regression
Fixed the target issue but broke something that was previously working. Most common in refactor tasks.
Spec misread
Solved a different problem than what was asked. Particularly common when instructions reference an error message.
Silent assumption
Made a judgment call on an ambiguous task without flagging it. The biggest differentiator between tiers: Sonnet 4.6 surfaces ambiguity more reliably than Haiku.
Non-running
Returned code that doesn't execute: syntax error, missing import, or malformed output.
Methodology
How this was built
25 tasks across 5 categories, adapted from real developer patterns rather than contest puzzles.
Each task was run 3 times at temperature 0.2. A task is marked passed only if 2 out of 3 runs succeed.
Ambiguous tasks are graded on whether the model acknowledged the ambiguity, not on code output.
Two models compared: Claude Haiku 4.5 (fast and cheap tier) vs Claude Sonnet 4.6 (advanced tier).
Code is graded by running it against hidden test assertions using Python exec() in an isolated namespace.
v2 - what I'd build next
This eval is single-shot: one attempt, then stop. That's not how real Claude Code usage works. A v2 would wrap these same 25 tasks in a feedback loop - the model reads files, runs code, sees what broke, and keeps trying. That's a cleaner test of how coding agents actually behave in practice, without the infrastructure overhead of SWE-bench.
Other additions worth making: TypeScript tasks alongside Python, system prompt experiments to reduce silent assumptions on ambiguous tasks, and auto-scaling difficulty by model tier.