Twitxy Saved

Alexander Johansen

Verified account This account is verified. Learn more

@AlexRoseJo

CS PhD @Stanford || Ask me about NVFP4 training github.com/alrojo/fp4Attn
Stanford, CA alrojo.github.io Joined October 2015
870 Following 1.4K Followers 1.1K Posts

Latest posts 5

Pinned post

Alexander Johansen Verified account @AlexRoseJo
🥐 Hi everyone! FlashAttention's backward pass has the same zero-sum bug
@jon_barron
just fixed in softmax. With FP4 stochastic rounding it's wrong on every single draw. We fixed it in our ICLR'27 submission, CroissAttn and unlock e2e FP4 training.
0:14
Jon Barron Verified account @jon_barron Oct 23
Hi all, I have a new result I'm excited to share. It turns out that *all existing implementations* of softmax and cross-entropy (that's JAX, PyTorch, textbooks, etc) have a subtle bug --- two bugs, actually. A simple rewrite fixes both, and eliminates "slingshots" when grokking:
View quoted post on X
25 replies 101 reposts 1.1K likes 74K views View on X
Alexander Johansen Verified account @AlexRoseJo
If your research interest is building our infrastructure for LLMs I can highly recommend @nebius_science they have supported much of our research with frontier GPUs (B300s), places to give talks, blog post opportunities, and we're working with their engineers now to scale our results.
Nebius Science Verified account @nebius_science 1d
What’s @nebiusai researching, and how can PhD students and professors get involved?

🎓 Join our Nov 23 webinar on our work in inference optimization, post-training, and agentic systems, and the new Nebius PhD Fellowship.

Sign up: nebius.science/events/webinar…
View quoted post on X
Alexander Johansen Verified account @AlexRoseJo
I'd prefer democratizing AI. Starting with educating as many as possible through hands-on courses like deep-learning-alchemy.github.io @TristanThrush
deep-learning-alchemy.github.io Stanford CS 312 | Deep Learning Alchemy Official course website for Stanford CS 312: Deep Learning Alchemy (Fall 2026), including logistics, schedule, assignments, and course materials.
Peyman Milanfar Verified account @docmilanfar Nov 6
The race to build the best AI is irrelevant at this point. It's now all about who scales affordable access to the most people.
View quoted post on X
Alexander Johansen Verified account @AlexRoseJo
If you want to learn more on how to train LLMs in NVFP4 come to my presentation at @nebius tomorrow 9AM PST! nebius.science/events/webinar…
nebius.science Training transformer attention entirely in FP4 with CroissAttn Follow how AI and science co-evolve — through the lens of the cloud powering it.
Soumith Chintala Verified account @soumithchintala Nov 5
Our v2 small model is here.
I've been using it daily for a variety of things.

The pre-training ran for 68 hours on GB300s. Post-training was 17 million rollouts over 9 days.
Entirely pre-mid-post trained in NVFP4.

Available to train and fine-tune on Tinker right now.
Open weights next week after external safety testing is fully complete (cant skimp on that).

Continuing to decentralize and democratize intelligence...
View quoted post on X
1 replies 4 reposts 36 likes 2.2K views View on X
Alexander Johansen Verified account @AlexRoseJo
Open source, open dataset, open training run, and even open scaling runs. Ontop of CS336 and CS312 @stanfordnlp is educating the world on how to develop frontier LLMs.
It’s done! Marin’s Mochi 535B-A23B finished its main pretraining phase this morning. 18T tokens! Long context this weekend, SFT over the next week or so. Hoping to release checkpoints next week. Some EU paperwork first. mumtracker.exe.xyz/hero-run-535b
View quoted post on X