Twitxy Saved

Jordan Sassoon

Verified account Aleph Alpha

@resiliem

Pre-training Efficiency @Aleph__Alpha
Germany Joined October 2023
127 Following 691 Followers 67 Posts

Latest posts 3

Pinned post

Jordan Sassoon Verified account @resiliem
What is not covered in the report is how we scaled model training to be efficient on 768 B200 GPUs.

I explain it in detail here: aleph-alpha.com/en/blog/scalin…
A how to guide to scale up model training without wasting your GPU hours. Even if you only have 1 or 2 GPUs and cannot scale, our optimisation philosophy can still make your GPUs go brrr.

I couldn't explain ALL the details of our method in the blogpost, since the big launch came after. You can connect the dots now, the 30B MoE architecture was in fact Kolibri Origin all along.
aleph-alpha.com Scaling Pre-Training in Practice: A Hierarchical Approach — Aleph Alpha Taking a 30B-A3B MoE model from 16 to 512 B200 GPUs while achieving near-linear scaling and 35.3% MFU: how we shrink the search space at small scale before increasing the GPU count.
Aleph Alpha @Aleph__Alpha Oct 3
Tech report is available: aleph-alpha.com/downloads/tech…
View quoted post on X
15 replies 76 reposts 737 likes 177K views View on X
Jordan Sassoon Verified account @resiliem
MoE load balancing was a key element to stable training of the Kolibri family. The team spent significant time developing custom solutions to this problem.

We, the pre-training efficiency guys, are very grateful for this. Token routing imbalances meant some GPUs crunched more data than others, creating wait times. You can see this resulted in a dip at the beginning of training, as the load balancing had not yet smoothed out routing.

After the initial dip, we maintained stable, high throughput of ~16k tokens per second per GPU thanks to the custom load balancing techniques.
Pit Neitemeier Verified account @PitNeitemeier Oct 15
We worked out efficient distributed calculation of global quantiles for quantile MoE load balancing. Additionally we found a better surrogate for straight through estimation of load errors that outperforms the common GShard loss for local balance. Check it out!
View quoted post on X
0 replies 3 reposts 48 likes 2.2K views View on X