Gradient

DeepSeek V4.1 Flash (1017) solved 8/17 Zero2Repo tasks with dsh, at $3.18 in reported API spend across the full suite.

OpenCode scored 7/17 and Claude Code, 5/17.

Costs are reported API spend across all 17 tasks.

Check out more findings: zero2repo.ai

SWE-bench Verified has become saturated. Seven models scored 95–97% (Vals AI, Sep 2026). But how well can coding agents build complete software projects from scratch?

Zero2Repo is a Gradient Data research initiative building hard, verifiable tasks and training data for whole-project development.

Agents start with a full spec, an interface contract and an empty workspace. They build the entire repo without reference code, then submit it for hidden deterministic testing. Every test must pass for a task to count. No LLM judge is used.

The first release includes 17 tasks derived from real open-source projects across Python, TypeScript, Go, C, C++ and Zig.

Across 18 model–harness configurations, scores range from 1/17 to 15/17. The top 10 span 47 percentage points, compared with 5 points on SWE-bench Verified.

No configuration has completed all 17 tasks. This matters for self-iteration: small unresolved issues can persist and compound as agents continue developing a project. Reliable self-evolution remains an open research challenge.

We’re expanding the task collection and looking for contributors to suggest seed repositories, develop tasks and review requirements and hidden tests.

More cases are coming.

👉 zero2repo.ai

👉 github.com/Gradient-Data/…

Over the past year, we’ve worked on Echo and Logits to make AI post-training more efficient and accessible.

Gradient Data now extends that work with verifiable environments, real-world trajectories and evaluation systems for more capable and reliable agents.

gradientdata.ai

A few research milestones worth looking back on as September comes to a close.

Our team and collaborators have been working across model infrastructure, agents, and evaluation, from making RL more efficient to testing whether AI systems actually do what they intend.

🧵

More capable models require more routing decisions, and better ways to benchmark how routers execute.

That’s what TwinRouterBench is built for.

Congrats to the team on the NeurIPS 2026 acceptance.