Chris McCormick

(1/3) - Here's how my September "speedrun the backward pass" project (decoder-rtx) played out.

Halfway through the month, a swarm of AIs run by rekursiv.ai set a new record on almost exactly this task, and porting their techniques is really what got us down to 12 minutes.

Their model uses giant n-gram tables, and needs a 141GB H200. We got it all to fit on the 96GB RTX 6000, though, by inlining the table optimizations into the backward pass so we could forgo 22.5GB in gradient buffers.

I've come to really like the architecture, and wrote it up as a blog post, published as the project README: github.com/chrisjmccormic…

A couple highlights are in this thread.

Big stack of improvements in baseline 4 cut the time by 40%.

Also, this "gpt-2 small" model is now 7 billion parameters trained on 350 million tokens. Ha!

Two particularly fun changes:
- Dropped .grad buffers from the hash tables and *inlined the optimizer* in their backward math.
- Added a 2M row tri-gram stats table (tallied from dataset, frozen), projected and added to values.

Some big efficiency gains in baseline 4:
- Learned you can run max-autotune once and preserve / ship the results alongside the model. Huge.
- Fable found "the right spelling" of ReLU^2 backward so Inductor would fuse it correctly.

Several other ideas were lifted from / inspired by an auto-research run published by rekursive.ai last week: x.com/hyperparticle/… (worth checking out, the results are more interesting / innovative than just hyperparameter tuning)
What I borrowed:
- Batch size of 192K
- Added gates to x0 lambdas
- Two huge bi-gram hash tables:
- 2M rows, added to x0
- 1M rows, added to values
- To optimize these 5.25B parameters: sparse, no first moment, and only per-row magnitude.

Unlike the auto-research run:
- Codebase is still ~clean / not Clauded-up 😜
- No FP8, custom ops / triton

PR: github.com/chrisjmccormic…

This is the last week! There's more to lift from the rekursive run and modded', but I'm going to try and focus on innovations in backward.

It's got a really cool symmetry!

Managed to shave about 70 seconds for this week's baseline, so we're just under 26 minutes.
Code's published in a pull request here, but I'll need another day to finish the write-ups:
github.com/chrisjmccormic…

Let's speedrun the backward pass.

To encourage more explanation of backward, I've set up a single-file, single-GPU challenge for the month of September.

The target GPU is the 96GB RTX Pro 6000, and the pipeline currently runs in 30 minutes end-to-end.

The handwritten backward pass + single GPU setup create interesting new opportunities.

And what better way to test out the new Fable 5.1 or GPT-6 this weekend than to tell them "see what you can do to speed this up." 😁

I won't be running the typical individual PR-based leaderboard--I'll be assembling a new baseline every Monday, through September 28th, and crediting everyone whose contributions make it in.

Check out the README for more:
github.com/chrisjmccormic…