Twitxy Saved

Spencer Frei

@sfrei_

Research Scientist at @GoogleDeepMind
San Francisco spencerfrei.github.io Joined May 2019
1K Following 3.2K Followers 338 Posts

Latest posts 5

Spencer Frei @sfrei_
This was a fun one to work on with Bruno! Given a scaling ladder and a fitted power law, how far is its fitting loss from the best achievable one? And how much do extrapolations of fits at least as good vary? Our paper gives provable answers to both
Bruno Mlodozeniec Verified account @brunorganised 4d
New paper with @sfrei_: "Ambiguous Fits"

To estimate a scaling law, you train a ladder of models, evaluate them, and fit a curve. The last step is usually treated as routine. It turns out this step has issues: curves that fit equally well with standard approaches can make very different predictions! 🧵
View quoted post on X
1 replies 2 reposts 9 likes 1.3K views View on X
Spencer Frei @sfrei_
First blog post: "Quantifying Pretraining Variance"

Data and init seeds are familiar sources of run-to-run variance in LLM training. But floating point arithmetic order (e.g. from sharding differences) also affects things, and we find these effects are nearly as large!
7 replies 11 reposts 126 likes 8.4K views View on X
Spencer Frei @sfrei_
There's at least four dimensions along which AI research progresses: 1) software engineering speed+execution, 2) algorithmic innovation, 3) hardware innovations, 4) quantity of hardware used for R&D. Agents so far have been helping a lot on 1 & 2, maybe 3, not yet on 4