MLG alumni 🤝 Twitter beef
as is tradition.
as is tradition.
I’m usually a big fan of a @giffmana takedown, but “this is well known” misses the core new things in our work:View quoted post on X
The 2022 paper is super relevant and we should have cited it. Agreed the E/α ambiguity in scaling laws is itself known, the paper doesn't claim to discover it. But the linked works don't give you a way to answer the questions: for a given ladder, is my fit actually near-optimal? How wide is the range of predictions from all near-optimal fits?
We contribute both:
1) Algorithms that provide provably guaranteed answers to those questions.
2) Using these algorithms, we quantify how suboptimal the fits of a standard approach people rely on are (Chinchilla Approach 3), and how much of a difference that can make when using the resulting scaling law for prediction.