Intelligence is solved. AI labs are shifting their attention to the next: proactive intelligence. AI that knows when to jump into your day-to-day.
Today, we're releasing the Agent Initiative Benchmark: a proactive intelligence test that measures 12 models' ability to solve real-world tasks in a carefully curated synthetic workspace, backed by a robust 6-year data blueprint.
Check out our founder Jakub's key takeaways for agent builders.
Today, we're releasing the Agent Initiative Benchmark: a proactive intelligence test that measures 12 models' ability to solve real-world tasks in a carefully curated synthetic workspace, backed by a robust 6-year data blueprint.
Check out our founder Jakub's key takeaways for agent builders.
What鈥檚 the best model for proactive agents?View quoted post on X
Today, we鈥檙e introducing the Agent Initiative Benchmark. We tested 12 frontier models on realistic workplace tasks inside a synthetic company with 6 years of highly detailed history: it has a financial model, CRM records, past-due invoices, Linear tickets, and real-life events affecting company development - competitor product announcements, SVB collapse, etc.
We used the company鈥檚 Slack to test how a proactive agent listening in on each conversation would assist the team in day-to-day tasks: from stepping in to align initiatives across teams to warning the team not to use an unapproved customer logo.
Highlights:
- 6.1 Sol and Astra from @OpenAIDevs beat Anthropic's latest Fable 5.2, with Sol crushing it at both performance and cost ($0.11 per task vs $1.87).
- The most expensive models aren't the best: Fable 5.1 is 430x more expensive than GPT-6 Luna, with similar performance.
- Most models try to help even if they're not needed. Interestingly, in these failure cases, they spend more time thinking, and the unnecessary work takes about 80% more time and money than if they stayed quiet.
If you鈥檙e building agents, keep reading for more tips on which model is best for your use case. Detailed breakdown, the open-source dataset, and benchmark code below: