Twitxy Saved

Matt Beton

Verified account This account is verified. Learn more

@MattBeton

ml systems & algos | prev @exolabs, maths @Cambridge_Uni
london mattbeton.com Joined February 2015
257 Following 2.7K Followers 270 Posts

Latest posts 5

Pinned post

Matt Beton Verified account @MattBeton
Training an LLM on 8 M4 Mac Minis

Ethernet interconnect between Macs is 100x slower than NVLink so Macs can’t synchronise model gradients every training step.

I got DiLoCo running so Macs synchronise once every 1000 training steps using 1000x less communication than DDP
166 replies 454 reposts 6K likes 555K views View on X
Matt Beton Verified account @MattBeton
LLMs are fantastic for implementation, but they still struggle to produce leap-of-thought ideas that are needed for pushing frontiers.

Our contribution moves up the abstraction stack, but there is still a need for mathematicians and problem solvers.
Deven Pietrzak Verified account @DevenPzak Sep 29
I really enjoyed working on this project. I believe the new flop-aware paradigm of nanoGPT thinking is useful, and it seems helpful in a compute optimization context like this one. It seems like a simple and logical conclusion of optimization: figure out what’s actually needed, and use that. The process was actually fun.

One note regarding this record, as noted in the blog. Some of the techniques that scale to frontier pretraining have been redacted from this record. Most importantly, ANVIL III, the significantly improved version of the ANVIL optimizer lineage, has not been included in this record, as it is proprietary to us at @hyperstition_cc. It replicates Muon’s speed but beats it by 20-28 millinats of loss at all model sizes 124M-1.2B and training depths (up to 8x-Chinchilla), with remarkably consistent hyperparameters and extremely high SNR, as compared to Muon. You can learn more about ANVIL III and Feather-1.7B, the model we trained using internal architectural changes and ANVIL III, here:
hyperstition.cc/cutting-pretra…

Using ANVIL III (which allowed me to cut steps) improved the speedrun results by another 0.7 seconds, with lower net loss. This alone proves there’s still more room to improve this record.

One final thing that I’ve had many people ask about is how much of a role autoresearch played in my work. The answer is almost none. While almost all of the implementation was done with Claude Code, I did the ideation almost fully by myself. I spoke about this extensively in a conversation with @METR_Evals.

We had a cluster of H100s while I was working on the record. At night or when I stopped working on it, I’d send out several agents to try to hill-climb the speedrun and improve the record. In total, the agents made shockingly little progress, despite spending more time in aggregate than I did. No matter what I told them, it seemed difficult to get them out of local minimums, stuck performing futile tuning of a method that saved 0.1 seconds and cost ~5 millinats (clearly a bad trade), or just tuning hyperparameters for hours on end trying to make some small change. I saw similar phenomena as those described in this METR report:
metr.org/blog/2026-07-2….

However, as I also discussed with METR, I don’t think the agents lack the knowledge to do this. Instead, I think a lot of it comes from a personality issue with current LLMs. They have very little belief that large jumps like this are possible, or they approach it with the wrong paradigm. Autoresearch was not very helpful in my work, and, from personal experience, I don’t think this current generation of publicly available models, without special harnesses, will do very high-quality autoresearch. However, I think the capability and knowledge is already here, which implies that we could be significantly closer to true autoresearch than recent results would imply.

NanoGPT was a really fun challenge and it led to a number of valuable insights and processes. Finally, I’d like to give @Classiclarryd and @KellerJordan a huge thank you for all of their efforts maintaining the repo.
View quoted post on X
1 replies 0 reposts 13 likes 1.6K views View on X
Matt Beton Verified account @MattBeton
I built a 50k parameter language model to run on an 8-bit 6502 processor from 1975. It learnt to spell words and construct basic grammar!

The model is build on a Mamba backbone (attention’s KV cache not fit in 32KB of memory) with a character-level tokenizer.

The MOS-6502 doesn’t have a ‘multiply’ instruction - only 8-bit integer addition. I therefore used ternary weights to reduce matrix multiplication to a series of add/subtracts operations.

Technical write-up here: mattbeton.com/blog/bitnet-65…
Source code: github.com/MattBeton/BitN…
19 replies 21 reposts 194 likes 27K views View on X