Grigory Sapunov

1/I've been reading ML papers for 20 years.
PhD in AI. Google Developer Expert. CTO of an ML company. Wrote a book on JAX.
But in 2024, I admitted defeat: I cannot keep up with arxiv anymore.
So I built a system to do it for me. Here's what I learned 🧵

1/ In a QAT run on TPUs, an intermediate cast vanished inside a fused kernel, evaluating x - cast(x) to x - x = 0 across 131,072 values.

Splitting the kernel to inspect it spilled to memory, restored the cast, and made the bug vanish. 🧵

1/ Claude Opus 4.5 can wipe its conversation after every single shell command on SWE-bench Verified and still hit 70.2% resolution, matching its 70.0% full-history baseline while cutting peak context from 130K to under 9K tokens. 🧵

1/
Nobody at our company writes code by hand any more. At all. Six months ago it was most of our code.

I predicted this in 2017, when every “jobs AI will take” list had drivers and lawyers on it, never programmers. Notes from a year that changed everything 🧵

1/
A 109M text model trained with good views matched next-token prediction with 42x fewer samples on WIT-1024 (.109 at 21.6M samples vs .111 at 900M).

Scaled up, it reached .184 mutual 10-NN with DINOv2, outscoring Qwen2.5-72B (.154) and Llama-3-70B (.151). 🧵