ORO

The economic value of agents will soon come from running whole autonomous businesses, not just from finishing one-off tasks. When you run a business, the result of your own decisions arrive weeks later. You have to notice them, trace them to something you did, and act before they compound.

Today, we're excited to announce SMB Bench, where we measure all the frontier models' ability to observe changes in the environment and adapt accordingly.

We've been working hard behind the scenes and our excited to share our vision, thesis and progression. x.com/i/article/2118…

ORO changelog, Oct 18 to Nov 1:
- ORO Bench now tests agents in shops with their own products, tools and situations, with pass/fail task scoring
- New public practice world for local testing, graded the same way validators grade
- Validators need v3.0+, and SDK clients need oro-sdk / oro-ai/sdk 7.0.0+
- Clearer episode feedback and more complete inference cost reporting

check out our changelog and what we've been working on:

As models and harnesses get more capable, the depth of our RL environments needs to stay ahead of the curve.

See how, over the past month, we have gone from ShoppingBench - Alibaba Group (Wang et al., 2025) → ORO Bench → AssistantEval

Even better environments coming soon!

We're excited to open source all the tasks and the traces that we have access to from this benchmark.

Link in 🧵