Matthew Siper

Albert Gu Verified account @_albertgu · Oct 28

I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.

Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?

Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.

View quoted post on X

Ahmed Khalifa @Amidos2006 · Sep 9

Cellular automata are powerful but painful to author. Rules are a lookup table; neural CAs are more expressive but a black box.

What if each CA rule was just code that you can read?

Introducing Programmable Cellular Automata (PCA)

View quoted post on X

will brown Verified account @willcb · Aug 23

trained on prime intellect lab :)

this is nuts my mind is kinda blown

View quoted post on X