Twitxy Saved

Pit Neitemeier

Verified account Aleph Alpha

@PitNeitemeier

pre-training @Aleph__Alpha
Joined December 2020
377 Following 313 Followers 63 Posts

Latest posts 5

Pinned post

Pit Neitemeier Verified account @PitNeitemeier
We’re so back! The calls to train models in Europe again were getting louder and louder and we worked incredibly hard to make this happen. It was a pleasure to build these teams from the ground up! Now off to do some euromaxxing. Looking forward to everyone trying the model!
Aleph Alpha @Aleph__Alpha Oct 3
Small bird, fast wings, Kolibri is here.

78B parameters. 3.46B active. Up to 1M tokens of context. Built in Europe.

Now the weights are yours. Run it on your own hardware, under Apache 2.0.
View quoted post on X
22 replies 8 reposts 284 likes 15K views View on X
Pit Neitemeier Verified account @PitNeitemeier
We worked out efficient distributed calculation of global quantiles for quantile MoE load balancing. Additionally we found a better surrogate for straight through estimation of load errors that outperforms the common GShard loss for local balance. Check it out!
Aleph Alpha @Aleph__Alpha Oct 15
Expert load balancing is key to training Kolibri efficiently at scale.

We developed two new methods: Exact Quantile Balancing (EQB) for global balance and Load-Error Injection (LEI) for local balance. @JesseLiJiaze and @PitNeitemeier explain them in a new, visual blog post.
View quoted post on X
0 replies 1 reposts 18 likes 2.7K views View on X
Pit Neitemeier Verified account @PitNeitemeier
One of my favourite things to see here is how for DeepSeek at short seqlens we read the attention state more than once (dashed read line above solid total) due to cross-layer reuse. Past 16K, sparse attention takes over, so we read it less than once and the lines cross.
Aleph Alpha @Aleph__Alpha Oct 7
The total attention state controls how many sequences fit during decode and the state read per step impacts decode speed. Sparse attention decouples the two by reading only a subset. With state reuse across layers, the read state can surpass the total.
View quoted post on X
0 replies 1 reposts 13 likes 382 views View on X
Pit Neitemeier Verified account @PitNeitemeier
How many of your active parameters go to attention, and how many to the experts? In our new blog we derive how modern architectures allocate parameters and FLOPs, and what that means for attention state and long-context FLOPs.

Also: cute kolibris!
0:27
Aleph Alpha @Aleph__Alpha Oct 7
Every architectural choice has a cost. Choose where to pay it.

@alessio__serra and @PitNeitemeier derive from first principles how design choices allocate parameters, FLOPs and KV cache, compare open models, and share some of the tooling we used to design Kolibri.
View quoted post on X
4 replies 3 reposts 34 likes 1.8K views View on X