@dlwh
Marin's 535b is now >80% through pretraining! Here’s loss vs. our scaling-ladder forecasts. The sharp drop comes with the 4K→16K context switch, so we're kind of cheating the forecast. (This was planned.) We'll re-eval at 4k too. forge.coreweave.com/wandb/marin-co…View quoted post on X
background- our data has longer docs so intent was to launch at 16k like MAI. However our MoE kernel only supported 4k (high dropping), so we planned to extend after kernel was improved. Since that eta was unknown, scaling ladder was kept at 4k, with knowledge that scaling ladder would cease to apply once new kernel landed and context grew. Quite remarkably that and many other kernel improvements landed, raising MFU from 20 to 30 mid run.View quoted post on X
The forecast on Uncheatable was added mid run where its fcst starts, but we didn't calibrate its asymptote or fit it to the data from the run, so its quite off. The main value we got from it was to directionally estimate impact of the earlier datamix change. Paloma was the eval used to evaluate if we were 'on track'.
Memory caps tokens per batch, so 4x longer context is paired with 4x fewer sequences.
Marin's 535b is now >80% through pretraining! Here’s loss vs. our scaling-ladder forecasts. The sharp drop comes with the 4K→16K context switch, so we're kind of cheating the forecast. (This was planned.) We'll re-eval at 4k too. forge.coreweave.com/wandb/marin-co…View quoted post on X
