Twitxy Saved

Nimit Sohoni

Verified account Cartesia

@nimit_sohoni

Stanford, CA Joined May 2016
4.2K Following 2.1K Followers 136 Posts

Latest posts 5

Nimit Sohoni Verified account @nimit_sohoni
that's nothing, check out what we have going on at Cartesia
Jacob Coxon Verified account @hilbertspaess Nov 13
This slide from the Anthropic IPO roadshow was a big factor in my decision to leave. It’s an insane plot and I’m surprised they made it public.

It shows the compute efficiency of pretraining over time. The y-axis is a **log scale**: they are seeing continued superexponential algorithmic progress.

Mythos was not a one-time deviation, but a point on a superexponential. They could already retrain Mythos with a fraction of the compute originally needed.

It doesn’t strictly imply a software-only intelligence explosion- they have been spending more on R+D. But algorithmic improvements do not require ridiculous creative genius and are on the verge of being automated. You do the math.
View quoted post on X
5 replies 5 reposts 107 likes 4.4K views View on X
Nimit Sohoni Verified account @nimit_sohoni
Check out what I've been working on recently!

(In other words: Check out why I've been slow to respond to people's texts recently 🫠)
Albert Gu Verified account @_albertgu Oct 28
I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.

Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?

Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
View quoted post on X
1 replies 2 reposts 37 likes 1.1K views View on X