Haodong Wen

This account is verified. Learn more

@herrywen1

Cogito, ergo sum | 2nd-year Ph.D student @Tsinghua_Uni

hhhhhh-925.github.io Joined June 2022

528 Following 398 Followers 157 Posts

Latest posts 5

Pinned post

Haodong Wen @herrywen1 Oct 26

(1/9)Suppose you're training an LLM on web data with limited math data. Should you place all your math data at the end of training, or mix it with web data earlier? The answer flips with the learning rate (LR) schedule!

At fixed compute and mid-training (MT) token budgets, earlier mixing gives lower MT loss when LR decays throughout training. With constant LR, placeing all MT data for the end works better.

We then vary when LR decay begins. The best tested strategy: keep LR high after MT data enters, then decay near the end.

See our new paper: "Data Schedules and Learning Rate Schedules Should Be Designed Together" !

0:56

Haodong Wen @herrywen1 6d

Nice one! If you believe LLM pretraining occurs at the EoS, understanding this under Muon is definitely a crucial piece of the puzzle!

cider Verified account @jeffreycider 6d

we discovered the edge of stability for muon
(and also every other non-adaptive optimizer)

we call it bubbleforce.

also we shaved off 100 iterations from the nanoGPT muon record (-3.2%)

View quoted post on X

Haodong Wen @herrywen1 6d

native FP4, interesting

Thinking Machines Verified account @thinkymachines 6d

Introducing Inkling 2 Small. Scoring 31 on the AA Intelligence Index, it's a big jump over its predecessor.

Inkling 2 Small is available for fine-tuning and inference on Tinker today, with full weight release planned soon.

thinkingmachines.ai/news/inkling-2…
View quoted post on X

Haodong Wen @herrywen1 Nov 4

Good one! It's like recipes for multiepoch is always underexplored. To my knowledge, tons of efforts on this. Perhaps I will organize the content related to multiepoch/data reusing under this thread.

Wenhao Chai Verified account @wenhaocha1 Nov 3

n-gram dominates recent pre-training speedruns, however, it totally sucks under multi-epoch setup. small vocabs survive.

larger tokenizers suffer more multi-epoch regret, the extra loss from repeating data instead of training on unique data.

Blog: wenhaochai.com/blogs/vocab-mu…

View quoted post on X

Haodong Wen @herrywen1 Oct 26

see our series for the training recipes in pretraining！

a scaling law that maps LR schedules to loss curves
(multi power law)
arxiv.org/abs/2503.12811

then studied how data reuse changes scaling across epochs arxiv.org/abs/2511.13421

how LR decay can weaken the benefits of data curricula. (CMA)
arxiv.org/html/2511.1890…

arxiv.org A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules Training large models is both resource-intensive and time-consuming, making it crucial to understand the quantitative relationship between model performance and hyperparameters. In this paper, we...

Haodong Wen Verified account @herrywen1 Oct 26

(8/9) This is a new exploration in a line of work we've been building on training schedules!

We began with a scaling law that maps LR schedules to loss curves, then studied how data reuse changes scaling across epochs and how LR decay can weaken the benefits of data curricula.

Now, we find that changing only the LR schedule can flip which data mixing schedule gives lower loss!
View quoted post on X