Twitxy Saved

Keller Jordan

Verified account This account is verified. Learn more

@kellerjordan0

CIFAR-10 fanatic
Pretraining @OpenAI OpCo LLC. Personal opinion poster
San Francisco Joined March 2016
486 Following 20K Followers 1.7K Posts

Latest posts 5

Keller Jordan Verified account @kellerjordan0
Seems like an interesting new speedrun

(I don't know whether it's inevitable, but this is a good way to find out.)
Samip Verified account @industriaalist Oct 12
Replacing backprop with zeroth-order optimization is inevitable.

Introducing Noise: a speedrun for pretraining without backprop.

Last week we presented Dust, the first method to approach backprop with SGD for pretraining transformers. We think another 1,000x improvement on top of it is possible, and it will require *co-designing* credit assignment, optimizers, architectures, and eventually, hardware.

We built Noise to iterate on this faster.
View quoted post on X
2 replies 6 reposts 140 likes 8.2K views View on X
Keller Jordan Verified account @kellerjordan0
I agree. Humans indeed uncover “depths of meaning” from data that seem to be inaccessible to machines.

In researcher-speak, it’s called data efficiency: Humans attain useful intelligence in only ~200M words, many OOMs faster than machines.

This difference between humans and machines isn’t necessarily a problem economically, because the high cost of training an economically useful machine can be amortized over many cheaply cloned inference workers.

But at a planetary scale, it could become a problem in terms of population dynamics or memetic entropy.

Large clonal swarms of insects are known to be vulnerable to viral collapse. A global population of intelligences which lack the human “regularization stack” — individuality and mortality — could yield unstable or inefficient global dynamics, similar to training an MoE neural network with poor load balancing.

Rather than being a matter of “good vs evil,” placing some limitations on the proliferation of machine intelligence could be necessary simply in order to prevent global “loss spikes.”
Pope Leo XIV Verified account @Pontifex Oct 8
Intelligence must draw on memory. The mind must not simply compile data — as an algorithm now does more quickly than we can — but recall lived experiences, which contain depths of meaning that only the human soul can recognize, and connect more deeply with the goodness of these experiences. Everything that is good begins “with the Spirit” and thus possesses a breath, a dynamism and a sense of communion that, once experienced, are engraved upon the heart. vatican.va/content/leo-xi…
View quoted post on X
10 replies 10 reposts 119 likes 9.7K views View on X
Keller Jordan Verified account @kellerjordan0
What a record by @DevenPzak! I’m honored that the speedrun continues to attract efforts of this quality.

In 2024, when the SOTA was 3min, @samsja19 asked where I thought the floor was. I said— maybe 2min?

It’s amazing how far it’s gone, thanks to @classiclarryd’s stewardship.
Larry Dial Verified account @classiclarryd Sep 28
New historic NanoGPT record at 39.9s (-27.7s) from @DevenPzak , obliterating the prior record of 67.6s!

This record introduces a new paradigm of thinking to NanoGPT: instead of optimizing matmuls or adding more expressive operations, optimize at the individual flop level with incredibly clever engineering and ML judgement. If a flop is low value on a particular step, skip it.
Specifically:
-(~8s) Sampled softmax. If a token doesn’t appear in a batch, skip its lm_head fwd/bwd some fraction of the time.
-Sparse values. Only run an optimizer step for ngram embeddings that occurred in the batch. Set beta1 to zero to enable this. Beta2 is applied retroactively when the row is later used.
-Sparse updates. Only update ngram and value embeddings once every 4 steps instead of once every 2.
-Sparse communication. Shard the n-gram table across GPUs, and only pass the rows receiving updates on each step.
-Sparse optimizer states. For the n-gram table, reduce from 2 floats in Adam optimizer per param, to 1 float per 768 params.
-Hand-rolled flash attention for 64 dim heads.

There are several additions that add accuracy too:
-(~4s) EMA during last 300 steps, combined with lifting final_lr to 0.3 instead of 0.15.
-(~1s) A new optimizer, Anvil2, which expands muon via a second tracked momentum buffer, improves the ortho coefficients, and modifies the cautious weight decay application.
-A couple additional dynamic skip connections in the network.

The most striking consequence of the ‘flop aware paradigm’ is you can grow parameters arbitrarily large,
only limited by the available memory, since you can selectively choose how to expend flops on those parameters on each step. NanoGPT has kept active parameters below 124M, but total is unbounded, and has grown to 640M through embedding sparsity over the last year. This PR takes that to its logical conclusion on the 8xH100, scaling up to 65B sparse embedding parameters, which accounts for 25% of the PR’s gains. At frontier scale, where one is not bounded by an 8xH100, one could imagine where this paradigm could lead.
github.com/KellerJordan/m…

As this was a very notable PR, I spoke with Deven for an hour to learn how he did it. Here’s his story on the changes: hyperstition.cc/training-nanog…
View quoted post on X