Keyu Wang

Larry Dial Verified account @classiclarryd · 1d

Sharing the thought process behind the architecture for Marin 535B-A23B, which finished pretraining on 18T tokens last week: openathena.ai/blog/marin-535…

Earlier this year the Marin project had 6 full-time employees at Open Athena, a mix of AI veterans and people newer to the space, like me. I learned a lot from the team as an IC. Since then we got our B200 GPU stack running in a month, scaled up 300x on GPU to a 535 billion parameter model, publicly forecasted the run, live-streamed the entire thing, and made it through pretraining slightly better than predicted with zero instabilities.

In the blog I call out specific contributions on architecture- but the broader gratitude is to the ecosystem. 2 years ago this would have been inconceivable on a first large scale attempt. Now it’s possible, in large part due to published work from Llama, DeepSeek, Qwen, Gemma, Olmo, Cohere, Nemotron, Arcee, Inkling, Mistral, Microsoft AI, Kimi, Muse, academia, and many others. The recipes are out there. A small team with big dreams (and some funding) can train a big model. (And if you have doubts on that, you can copy our fully open recipe from data to code).

Of course, training a model does not produce a binary outcome. I am proud of how far we’ve come and optimistic about how much more there is to explore! The team has grown recently and will be taking the model through SFT and RL soon, sharing the findings along the way.

View quoted post on X

Benhao Huang Verified account @huskydogewoof · 12h

Some unpopular-but-true notes on pretraining research:

→ 𝗗𝗮𝘁𝗮 𝗿𝗲𝗽𝗲𝘁𝗶𝘁𝗶𝗼𝗻 is a very common practice.
→ 𝗢𝘃𝗲𝗿-𝘁𝗿𝗮𝗶𝗻𝗶𝗻𝗴 (~1000 TPP) is a very common practice.
→ In the long run, 𝗱𝗮𝘁𝗮-𝗲𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁 architectures and algorithms will matter more than (currently) 𝗵𝗮𝗿𝗱𝘄𝗮𝗿𝗲-𝗳𝗿𝗶𝗲𝗻𝗱𝗹𝘆 ones.
→ 𝗙𝗶𝗻𝗲𝗪𝗲𝗯-𝗘𝗱𝘂 is a bad dataset for testing architectures; it can be misleading.

View quoted post on X

Cooper Leong Verified account @cooperleong22 · 2d

太长不读：由于具有尺度不变性的归一化层具有权重缩小 c 倍、梯度会大 c 倍的性质，训练后期学习率衰减与权重衰减的相互作用会导致这些模块梯度变大。经过一番推导后，每步实际的权重衰减幅度应该按（当前 lr / peak lr）的平方缩放。

View quoted post on X