On this slow news day, we just launched Stripe evals to measure how well coding agents build real Stripe integrations.
A few cool (to me) learnings:
β’ Our plugin helped every harness/model pair we tested. All 14 configurations scored ~3β8 percentage points higher with Stripeβs recommended skills and tools.
β’ Agents searched less and wrote fewer tests with the plugin. Web searches dropped 96% and (often extraneous) test code dropped ~40%, while scores improved.
β’ We used another agent to role-play the developer. The role-player starts by sharing its imperfectly-stated goals, asks follow-up questions, and works through a few rounds of conversation with its coding agent.
β’ The last mile is hard. Agents satisfied ~78% of criteria on average, but only ~29% of runs met every required criterion.
Results and methodology: stripe.dev/evals
A few cool (to me) learnings:
β’ Our plugin helped every harness/model pair we tested. All 14 configurations scored ~3β8 percentage points higher with Stripeβs recommended skills and tools.
β’ Agents searched less and wrote fewer tests with the plugin. Web searches dropped 96% and (often extraneous) test code dropped ~40%, while scores improved.
β’ We used another agent to role-play the developer. The role-player starts by sharing its imperfectly-stated goals, asks follow-up questions, and works through a few rounds of conversation with its coding agent.
β’ The last mile is hard. Agents satisfied ~78% of criteria on average, but only ~29% of runs met every required criterion.
Results and methodology: stripe.dev/evals