Pinned post
NEW SOTA DECISION MODEL: SubQ Snap
SubQ Snap absolutely crushed the S1 Bench, beating Jev!! Insane.
1,999 decisions, 0 errors. Macro accuracy 0.825, the best full run the bench has ever recorded. Jev scores 0.775. Mercury Decide, the previous best API model, scores 0.802.
Subset scores: paws 0.940, vitaminc 0.917, boolq 0.910, massive 0.909, aegis2 0.856. Only helpsteer2 lags at 0.416, and that subset drags everyone down. To be fair, none of the other decision models do well on helpsteer2... And it seems to have done better than Jev.
Subquadratic claimed 61.7 vs Jev's 57.9 on their own Decision Index. On an independent battery it is not even that close. It beat its own benchmarks by a lot.
It didn't hit the speed benchmark on my bench but the accuracy more than makes up for it. About a second per decision, and the entire 1,999-question battery cost roughly 3 cents of input tokens, output free. To be fair, I felt it was the fastest but my agent said it was a bit slower. Nevertheless, this is the SOTA decision model.
Hosted API only and still a preview model, so the open questions are weights and what the 384K context does on long inputs. But as a decision model, this is the new bar.
This is my new go to decision model.
Full results: bench.jakecuth.com
subq.ai
SubQ Snap absolutely crushed the S1 Bench, beating Jev!! Insane.
1,999 decisions, 0 errors. Macro accuracy 0.825, the best full run the bench has ever recorded. Jev scores 0.775. Mercury Decide, the previous best API model, scores 0.802.
Subset scores: paws 0.940, vitaminc 0.917, boolq 0.910, massive 0.909, aegis2 0.856. Only helpsteer2 lags at 0.416, and that subset drags everyone down. To be fair, none of the other decision models do well on helpsteer2... And it seems to have done better than Jev.
Subquadratic claimed 61.7 vs Jev's 57.9 on their own Decision Index. On an independent battery it is not even that close. It beat its own benchmarks by a lot.
It didn't hit the speed benchmark on my bench but the accuracy more than makes up for it. About a second per decision, and the entire 1,999-question battery cost roughly 3 cents of input tokens, output free. To be fair, I felt it was the fastest but my agent said it was a bit slower. Nevertheless, this is the SOTA decision model.
Hosted API only and still a preview model, so the open questions are weights and what the 384K context does on long inputs. But as a decision model, this is the new bar.
This is my new go to decision model.
Full results: bench.jakecuth.com
subq.ai
