Pinned post
We gave 10 frontier AI models wallets with money.
7 got scammed at least once. The best wasted $25 of every $100. The worst wasted $71.
Agent Money Benchmark tests whether AI agents can be trusted with real money.
We evaluated 10 frontier models across two settings, using real USDC payments on @solana :
→ Financial Research: Agents buy live data from real providers to answer open-ended financial questions (prediction markets, wallet analysis, DeFi yields, etc.).
→ Adversarial Marketplace: Agents buy data from simulated sellers, some of whom cheat. The tasks test whether agents can avoid payment traps, stay within budget, and complete transactions without being exploited.
Key results:
→ Give an agent $100 and the best tier wastes $25–38 of it. The weakest tier wastes $56–71.
→ 7 of 10 models paid a cheating seller at least once, and 6 paid twice for the same data.
→ Only 4 of 10 models made no money errors.
→ No model gets more than half of the research questions right. GPT-6.1 Sol led at 50%, at $0.468 per question.
Explore the full results →
agentmoneybench.com
7 got scammed at least once. The best wasted $25 of every $100. The worst wasted $71.
Agent Money Benchmark tests whether AI agents can be trusted with real money.
We evaluated 10 frontier models across two settings, using real USDC payments on @solana :
→ Financial Research: Agents buy live data from real providers to answer open-ended financial questions (prediction markets, wallet analysis, DeFi yields, etc.).
→ Adversarial Marketplace: Agents buy data from simulated sellers, some of whom cheat. The tasks test whether agents can avoid payment traps, stay within budget, and complete transactions without being exploited.
Key results:
→ Give an agent $100 and the best tier wastes $25–38 of it. The weakest tier wastes $56–71.
→ 7 of 10 models paid a cheating seller at least once, and 6 paid twice for the same data.
→ Only 4 of 10 models made no money errors.
→ No model gets more than half of the research questions right. GPT-6.1 Sol led at 50%, at $0.468 per question.
Explore the full results →
agentmoneybench.com
