Wu Lin

@LinYorker

Postdoctoral fellow at @VectorInst. ML PhD at UBC. Mathematical and computational structures for ML. Geometric and algebraic methods.

Orlando Florida yorkerlin.github.io Joined August 2020

41 Following 326 Followers 115 Posts

Latest posts 5

Pinned post

Wu Lin @LinYorker Jul 16, 2024

#ICML2024
Can We Remove the Square-Root in Adaptive Methods?
arxiv.org/abs/2402.03496
Root-free (RF) methods are better on CNNs and competitive on Transformers compared to root-based methods (AdamW)

Removing the root makes matrix methods faster: Root-free Shampoo in BFloat16 /1

Wu Lin @LinYorker Oct 14

I still remember the comparison between SGD and Adam a few years ago. Empirical results may not support that. So, wait and see just like the SGD vs Adam lesson.

jianlin.su @Jianlin_S Sep 28

Slow is Fast: Is Muon the end of Shampoo?
kexue.fm/archives/11917

Orthogonal Slowest Descent — from a convex optimization convergence bound — washes away any Kronecker preconditioner's eigenvectors, leaving only eigenvalues to balance the gradient's spectrum — leads to Muon.
View quoted post on X

Wu Lin @LinYorker Jun 12

On one hand, it is essential to tune baseline methods well on a model. On the other hand, it may be better to avoid using a model/architecture that has been modified and optimized for a single method for 1.5 years.

Konstantin Mishchenko Verified account @konstmish Jun 11

I just submitted a PR to modded-nanogpt with better hyperparams. With them, Muon can reach the target loss after 3250 steps instead of 3325.
Always tune your baseline well when doing research. Weak baselines can make any idea look promising
View quoted post on X

Wu Lin @LinYorker Apr 30

@_arohan_ Muon

Wu Lin @LinYorker Apr 19

@weijie444 Looks like a KFAC-based method with modern clipping? G(ZZ^T)^{-1} is known as the FOOF update arxiv.org/abs/2201.12250 while msgn() can be interpreted as "generalized (preconditioned) gradient norm clipping" arxiv.org/abs/2506.01913 .

arxiv.org Gradient Descent on Neurons and its Link to Approximate... Second-order optimizers are thought to hold the potential to speed up neural network training, but due to the enormous size of the curvature matrix, they typically require approximations to be...

Weijie Su Verified account @weijie444 Apr 16

We released "The Newton--Muon Optimizer" . We show that Muon is secretly an implicit Newton method, and use this insight to build a better one. 1/n

Paper: arxiv.org/abs/2604.01472
View quoted post on X