Pinned post
What is not covered in the report is how we scaled model training to be efficient on 768 B200 GPUs.
I explain it in detail here: aleph-alpha.com/en/blog/scalin…
A how to guide to scale up model training without wasting your GPU hours. Even if you only have 1 or 2 GPUs and cannot scale, our optimisation philosophy can still make your GPUs go brrr.
I couldn't explain ALL the details of our method in the blogpost, since the big launch came after. You can connect the dots now, the 30B MoE architecture was in fact Kolibri Origin all along.
I explain it in detail here: aleph-alpha.com/en/blog/scalin…
A how to guide to scale up model training without wasting your GPU hours. Even if you only have 1 or 2 GPUs and cannot scale, our optimisation philosophy can still make your GPUs go brrr.
I couldn't explain ALL the details of our method in the blogpost, since the big launch came after. You can connect the dots now, the 30B MoE architecture was in fact Kolibri Origin all along.
Tech report is available: aleph-alpha.com/downloads/tech…View quoted post on X