Ferdinand Kapl

Looped LLMs are everywhere right now! But why are they interesting? Do they model language differently than regular LLMs?

Grown and looped transformers are better sequence models!

With @kaitlinmaile & @oswaldjoh (@Google, Paradigms of Intelligence Team) and @stefanAbauer. 🧵

This project started with a literature search on "growing" work in modern vision pipelines, and I was surprised that there was not much, although there have been more recent works in language. Since I started my PhD by working on vision (object-centric?!) I was interested.

SOTA LLMs are trained as large monolithic models. But looping or depth growth can beat that. How?

Co-led with @AngelisManos and @TobiasHppe1.

Incredible collaboration with Paradigms of Intelligence Team @Google 🙌
@kaitlinmaile
@oswaldjoh
@ninoscherrer
and @stefanAbauer. 🧵🔽