Kobe (@kobe0938)

awesome work, built & run with harbor

harbor x runta

Guanlan Dai Verified account @guanlan 1d

We’re spending up to 100B tokens across 26,040 agent runs to answer one question: when you switch models, should you switch harnesses too?

Our v1.0 FrontierHarness leaderboard ranked harnesses on a single model.
V1.1 expands it to 21 harnesses × 5 models × 62 tasks × 4 attempts, 72x the size of v1.0.

Results are coming in live. Link below🧵

View quoted post on X

Kobe @kobe0938 6h

1. tb is popular:
"HLE, SWE-Bench Pro, DeepSWE, and Terminal-Bench saw universal adoption"

2. tb is actively maintained:
"Labs cycled through four versions of Terminal-Bench (2.0 → 4.0) in a single year"

tb follows semantic versioning (2.0 → 2.1 → 3.0 → 4.0). we don't just talk about continuous benchmarking, we enforce it, every single release.

every tb release also turns what i call the terminal-bench-harbor flywheel:

0. benchmarks: building and running harbor-format benchmarks (like tb) surfaces bugs, reward hacks, and gaps
1. harbor: every fix and missing feature lands here, e.g. regrade and stronger reward hack defenses
2. harbor hub: leaderboard releases, trajectory sharing and more stable hosted runs, e.g. error handling and auto concurrency control
3. and a sturdier stack brings more creators and teams with benchmarks like tb, running on harbor, releasing scores on harbor hub, which kick off the next turn

we'll keep looping and dogfodding the whole stack all the way to tb 20 ;)

0:10

Jason Wei Verified account @_jasonwei 1d

Created a "benchmark benchmark": a frequency distribution of which benchmarks were reported by the frontier labs in 2026: claude.ai/artifact/22cBP…

Some observations:
- HLE, SWE-Bench Pro, DeepSWE, and Terminal-Bench saw universal adoption
- Scale AI is behind two of those four (SWE-Bench Pro, HLE w/ CAIS), plus MCP-Atlas (#9 benchmark)
- Of the frontier labs, OpenAI is the most prolific benchmark producer (SWE-bench Verified, BrowseComp, GDPval); people don't seem to mind evaluating on a benchmark made by a competitor
- Anthropic has just one on the chart (BioMysteryBench) that basically nobody uses 🤔
- In academia, strong showings from Princeton (SWE-bench, CharXiv), Stanford (Terminal-Bench), Berkeley (CyberGym, Agents' Last Exam), and NYU (GPQA)
- DeepSWE saw incredible adoption, from first appearing in June to all 12 labs reporting it
- Labs cycled through four versions of Terminal-Bench (2.0 → 4.0) in a single year

View quoted post on X

Kobe @kobe0938 2d

tb4.1 is out with lots of task fixes + reward hack patches.

my favorite example: 4.0 verifier only passed models extruded from 2D sketches. agents started building the part from 3D primitives (gpt-5.6-sol’s approach, which might be smarter) instead and got failed for it. 4.1 accepts both.

0:11

Ryan Marten Verified account @ryan_marten 2d

Terminal-Bench 4.1 is here with updated verifiers for all tasks.

In this release, we focused on:
- resisting reward hacking
- reducing false negatives
- improving reproducibility

Re-grading the 4.0 leaderboard trials with the 4.1 verifiers, resulted in ~1% of rewards flipping.

Leaderboard scores remain within previously reported confidence intervals.

View quoted post on X

Kobe @kobe0938 2d

standardization is at the heart of harbor. for error handling, we now have one source of truth: error.policy, with 3 independent decisions:

1. should_retry: eligible for another attempt?
2. should_grade: run the verifier?
3. has_valid_reward: count the reward in evals/RL?

e.g.
• AgentTimeoutError: no retry, grade, reward valid
• SandboxBuildFailedError: retry, don’t grade, reward invalid
• ApiRateLimitError: retry, grade, reward invalid

an agent timeout is on the agent. a broken sandbox or a rate limit isn’t. separating them gives us fairer evals and cleaner training signals. only in harbor :)

0:12

Kobe @kobe0938 4d

This tweet changed everything.

11/07/2025 → 11/07/2026
Happy 1st birthday, harbor & tb2 🎂

1B+ Harbor rollouts
100T+ tokens/week through Harbor
Terminal-Bench on 64 model cards, the most-cited benchmark at model launches
10M+ Terminal-Bench runs
7.3M Harbor PyPI downloads in the last 30 days
550 contributors, 3,300+ merged PRs, 12k GitHub stars
5k+ on Discord
now tb2.1, tb-science, tb3 & tb4, harbor v1 on the way

it was the best of times. thank you all ❤️

Alex Shaw Verified account @alexgshaw Nov 7, 2025

Today, we’re announcing the next chapter of Terminal-Bench with two releases:

1. Harbor, a new package for running sandboxed agent rollouts at scale
2. Terminal-Bench 2.0, a harder version of Terminal-Bench with increased verification

View quoted post on X

Kobe

Latest posts 5