Twitxy Saved

Alex Ratner

Verified account Snorkel AI

@ajratner

@SnorkelAI / prev @uwcse @StanfordAILab – Interested in data management systems for machine learning, weak supervision, and impactful applications.
Menlo Park, CA ajratner.github.io Joined November 2013
786 Following 7.7K Followers 2.1K Posts

Latest posts 5

Pinned post

Alex Ratner Verified account @ajratner
I'm excited to announce @SnorkelAI's $350M Series E at $3.5B, led by @insightpartners and @S32_VC.

We've grown 18x+ in the last 12 months since launching our Data-as-a-Service offering, passing $375M ARRR this week.

As AI advances to superhuman capabilities, AI data & environment development must advance with it - and basic staffing and crowdsourcing approaches are not enough.

AI progress now requires deep research and technology work that combines human expertise with specialized AI in compounding ways. @SnorkelAI is building the RSI data engine and frontier data lab for this next phase.

We're honored to have the support of existing investors Addition, @lightspeedvp, @GreylockVC, @GVteam, P7, Factory, @WellsFargo, Walden Catalyst Ventures, and new investors @ThirdPointLLC, @MarchCPs, @BlumbergCapital, @AllegisCapital, @Frontlinevc, and @standard_vc.

–

@SnorkelAI started as a research project a decade ago at @StanfordAILab.

Our thesis was simple: AI progress would become increasingly data-centric – and therefore data development should be studied as a true research and technology problem, not just a staffing and crowdsourcing one.

Today, as AI capabilities verge on superhuman, building the data and environments to safely measure and train AI is becoming too hard for even the smartest human experts to do alone.

Only humans and AI agents, collaborating together in compounding ways, can meet the accelerating needs of the frontier, and keep humans in the driver’s seat of AI progress for decades to come.

At @SnorkelAI, we are building the data lab to define the shape of this new “Data 2.0” frontier, and the new paradigms of human-computer interaction needed to advance it.

Our key focus is building the RSI engine for data, where specialized AI models accelerate and improve human expert output, and in turn, scaled human supervision is used to continuously evaluate and improve these models – creating a powerful compounding loop to keep pace with an accelerating RSI frontier.

With this round of funding, we are also doubling down on our commitments to support data development for open benchmarking and evaluation (more news here soon!); an increasingly diverse ecosystem of general and specialized intelligence; and a path to safe, well-aligned AI built on robust training and evaluation data.

Data development will guide and drive the next stages of AI – and must do so in a human-centric, AI accelerated, open, diverse, and safe way. We are excited to support this mission in the next decade of research ahead at @SnorkelAI.

More thoughts here: snorkel.ai/blog/data-2-0-…
snorkel.ai Data 2.0 and the research era of AI data Announcing Snorkel’s $350M Series E financing at a $3.5B valuation, led by Insight and S32, with participation from Third Point, March, Blumberg, Allegis, Standard VC, Frontline, and existing...
72 replies 117 reposts 418 likes 93K views View on X
Alex Ratner Verified account @ajratner
This is an awesome perturbation analysis!!

We should be doing more work as a community to produce meta-analyses like these that measure robustness/overfitting in a transparent and rigorous way.

As a side note: it is also *baffling* to me how anyone could look at this, and come away with the one conclusion that "benchmarks are bad". This is a measurement of the degree to which *models* are overfit to a benchmark. That is controlled by the *model developer*. This is basic ML...

Note that *any* static benchmark can be perfectly overfit in the theoretical maximum, leading to results like these. Yes, some benchmarks will make this easier to do - which is why we should strive to make benchmarks more robust - but ultimately this is a property of models / model training.

I'm excited for the *more* of these meta-analyses - both to illustrate model robustness, and to inspire new more robust + dynamic styles of benchmarks that make overfitting more difficult!
Justin Lo Verified account @loresjpg 2d
Nice DeepSWE score! Does it survive a lowercase filename?

Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement.

Models clustered around 74% on DeepSWE end up 23 pp apart.

Good at coding, or good at DeepSWE? 🧵
View quoted post on X
5 replies 3 reposts 56 likes 3.1K views View on X
Alex Ratner Verified account @ajratner
Awesome work connecting end economic value to raw compute energy, and excited to support via @SnorkelAI open benchmark grants!!

Also: excited for @JonSaadFalcon to present at our @SnorkelAI Reading Group tonight- come stop by!
Jon Saad-Falcon Verified account @JonSaadFalcon 4d
AI is using more and more of the world’s energy. But how much economic value are we getting from AI?

We’re introducing Joules-to-GDP (J2G), a framework that measures how well AI turns energy into economic value and shows which tasks, models, and hardware give the best return.

Across 44 occupations in finance, engineering, law, administrative work, and more in @OpenAI's GDPval, we arrive at three findings:

1) The tasks with the highest economic value are not the tasks with the highest economic value per joule. For example, CAD design earns ~10% more per task than building websites, but the latter earns ~27% more value per joule.

2) The most powerful model isn’t always the best use of energy. Gemma 4 31B delivers 56% of GLM 5.2’s net value with 23% of the energy, for 2.5x the value per joule.

3) More chips don’t guarantee proportionate returns. Moving the same model from 1 to 8 NVIDIA H100s uses 3.2x the energy but cuts completion time by only 22%, with about the same value.

With @SnorkelAI, we’re also releasing GDPWorks, a benchmark for economically valuable tasks across 36 additional occupations.

Joint work co-led by @amirzeinaalii and @Avanika15, with @Mersad_Abbasi, @Jon_Hartley_, John Hennessy, @HazyResearch, and @Azaliamirh.

More details below 🧵
View quoted post on X
0 replies 11 reposts 33 likes 1.3K views View on X
Alex Ratner Verified account @ajratner
An important and good point! Ideally we should be judging based on output criteria:
- Quality (incl realism) according to detailed and properly factorized spec
- Complexity/difficulty
- Alignment with target usage/gap distribution
- Robustness to reward hacking / general rubric quality
- Etc

If the spec is sufficiently well defined, the input should purely be part of the how, not the what
Karen Zhou Verified account @karenzzhou Oct 29
"real envs > synthetic envs" feels like mostly 3 things:
- a complaint about sloppy synthetic data pipelines
- a way to make envs pricier without making them better
- synthetic task authors guessing the task distribution because they've never worked with real production traffic at scale

real envs ship very hackable tasks too. whereas a carefully built synthetic env that's realistic enough has all the same benefits as real envs, + you get to aim at one failure mode on purpose and isolate it from the random noise
View quoted post on X
2 replies 7 reposts 30 likes 2.9K views View on X