Twitxy Saved

Markus Zhang

@photon_mz

junior cs/math @stanford, intern pretraining @tilderesearch
Stanford, CA markusz.dev Joined August 2016
925 Following 535 Followers 519 Posts

Latest posts 5

Pinned post

Markus Zhang @photon_mz
Over the weekend, I managed to beat the NanoGPT world record by 0.3s, by untying Value Embeddings (VE) 😊

There’s a growing trend in sparse embeddings (Deepseek engram, Meta STEM), and @karpathy himself has extensively ablated VEs in nanochat
Larry Dial Verified account @classiclarryd Jan 29
New NanoGPT Speedrun WR at 99.0 (-0.3s) from @photon_mz, with an update from 3 to 5 value embeddings, enabling 1.5% fewer training steps! The trend of fewer steps with higher sparsity continues.
github.com/KellerJordan/m…
View quoted post on X
6 replies 2 reposts 22 likes 3.5K views View on X
Markus Zhang @photon_mz
Glad to contribute my second WR to the nanogpt speedrun, -0.4s!

I continued my work on VEs and found:
1. Later layers benefit from VEs more
2. Spherical Gaussian init is good
3. Gates should take input from both the VE and the residual X
Larry Dial Verified account @classiclarryd Feb 16
The NanoGPT Speedrun WR has broken below 90s, dropping from 92.1 to 89.8 from 4 recent contributions:
1. Tuned Kernels (-0.4s) from @.EmmetBicker on github & AI System Aster
2. Tuned Value Embeds (-0.4s) from @photon_mz
3. Sparse comms for bigram gradients (-0.3s) from @roeeshenberg
4. max_seq_len schedule and increased min lr (-1.2s) from @.dualverse-ai on github & AI System Station
View quoted post on X
Markus Zhang @photon_mz
in high school I was obsessed with making the perfect playlist, so I scraped all of public Spotify to train embedding models. I was hella dumb and had a vocab of 10M := 10M distinct songs, and got destroyed chatgpt “prompt
me a playlist”. Led me to obsession with chatgpt
λux Verified account @novasarc01 Jul 29, 2025
share a piece of lore about yourself getting into research
View quoted post on X