sathvik

Warp

@vikvang1

Semper Prorsum

DevEx Engineer @warpdotdev, Georgia Tech Alum, ex-Perplexity

vikvang.xyz Joined January 2021

1.7K Following 4.7K Followers 2.2K Posts

Latest posts 5

Pinned post

sathvik @vikvang1 Oct 29

3d game demos aren't the best way to judge a model's coding ability according to @theo. neither is SWE-bench, CursorBench, or TermBench.

instead, you should benchmark models against your own traces to find the best one for your tasks.

here's how the Warp team benchmarks various harness/model configs, which helped reduce cost / PR from $80 to $20 in our software factory (alongside uncovering a number of unintuitive findings!)

Demo:

4:51

sathvik @vikvang1 1d

lol

Garry Tan Verified account @garrytan 2d

@bb_fresh Going to change the description of gstack to an abstract software factory provider
View quoted post on X

sathvik @vikvang1 2d

In SF next week for AI Engineer! Who’s around? 😊

sathvik @vikvang1 3d

couple benefits to this approach we're taking @warpdotdev build out our mobile app:

1/ the more i work at this, the better the system gets over time
2/ the factory gets cheaper over time (we've went from $80 -> $20 per PR at Warp)
3/ you move at the speed of light: the factory generates a couple mockups, i choose from what i like best, maybe tag an teammate for input, and then ship it
4/ nearly every PR comes with its own computer use verified video + screenshots, so it's incredibly easy to grok what the PR did
5/ the factory doesn't improve from just my usage, but also learns from the learnings of other factories we have set up internally

the diff you see below was autonomously created with no input from me

Ben Holmes Verified account @BHolmesDev 4d

We built a mobile app for software factories, using a software factory, in one month.

I think our code review and verification process were a big part of that. We have agents:

- take computer use captures for every change
- post a /merge-risk breakdown with PRs so we know where to focus our attention
- complete rounds of adversarial code review when we move from prototype to PR

Excited to get this app in people's hands!
View quoted post on X

sathvik @vikvang1 3d

problems with today's benchmarks:

> they become outdated fast
> they are expensive to run
> many are gimmicks at best that don't communicate how useful a model actually is for YOUR workflows

instead, you should create custom benchmarks for yourself with your own tasks.

as a bonus, you can have a cloud agent automation that periodically curates eval sets for you (based on your own data), runs the benchmarks, shows you the results, and updates your entire factory system config on autopilot.

sathvik Verified account @vikvang1 Oct 29

3d game demos aren't the best way to judge a model's coding ability according to @theo. neither is SWE-bench, CursorBench, or TermBench.

instead, you should benchmark models against your own traces to find the best one for your tasks.

here's how the Warp team benchmarks various harness/model configs, which helped reduce cost / PR from $80 to $20 in our software factory (alongside uncovering a number of unintuitive findings!)

Demo:

View quoted post on X