Pinned post
Since compute grows faster than the web, we think the future of pre-training lies in the algorithms that will best leverage ♾ compute
We find simple recipes that improve the asymptote of compute scaling laws to be 5x data efficient, offering better perf w/ sufficient compute
We find simple recipes that improve the asymptote of compute scaling laws to be 5x data efficient, offering better perf w/ sufficient compute