Twitxy Saved

Ben Hoover

@Ben_Hoov

Building the next generation of energy models. @prefrontal_ai
London bhoov.com Joined February 2018
422 Following 2K Followers 320 Posts

Latest posts 5

Ben Hoover @Ben_Hoov
So the title of this work is severely understated

Sudoku sees anywhere from 100,000x - 215,000x speedup with CRAM

Here is a short vid showing just how many independent CRAMs we can train in the time it takes to train a single TRM on sudoku
0:17
Ben Hoover @Ben_Hoov 4d
Big reasoning speedups are here. 1,000× faster training and 280× less compute than any looped model. All done using a fully trainable energy from the 80s

Keep reading to hear how energy is the right paradigm for reasoning

With @HughBlayney @jcrwhittington @prefrontal_ai

🧵/10
View quoted post on X
6 replies 3 reposts 91 likes 12K views View on X
Ben Hoover @Ben_Hoov
Pushing the frontiers on long context modeling. All open source.

Great work team 🫡
James Whittington Verified account @jcrwhittington Nov 2
Today we are happy to share one of our first open research releases: Latent Sparse Attention (LSA), a step towards subquadratic attention that matches accuracy of full attention at ultra-long context.

Sparse attention attends to only a few tokens per query, but it has to find the right ones. At 10M tokens, that search takes 98% of the attention layer's time. LSA makes sparse attention up to 22.7× faster and full attention 72.4x faster.

With @kubicek_ales @caciolai @PengLiangzu @SukritSumant @Ben_Hoov @FraserGreenlee @RMBattleday @radim_urban @thoefler @jcrwhittington
View quoted post on X
1 replies 2 reposts 18 likes 1.4K views View on X
Ben Hoover @Ben_Hoov
Transformer looping = recurrence = naive weight sharing. A simple idea that works great

But look deeper. Can the recurrence be solving an optimization problem? This is the goal of understanding transformers as energy-based models. And it unlocks serious connections to physics 👇
Hayden Prairie Verified account @hayden_prairie Apr 15
We’ve been thinking a lot about scaling laws, wondering if there is a more effective way to scale FLOPs without increasing parameters.

Turns out the answer is YES – by looping blocks of layers during training. We find that predictable scaling laws exist for layer looping, allowing us to use looping to achieve the quality of a Transformer twice the size.

Our scaling laws suggest that for a fixed parameter budget, data and looping should be increased in tandem!

🧵👇
View quoted post on X
2 replies 27 reposts 191 likes 23K views View on X
Ben Hoover @Ben_Hoov
NRGPT shows how causal inference can be viewed as explicit optimization: next-token prediction becomes *energy minimization over token states*. Pretraining shapes the energy. Generation explores it

Excited to scale these ideas & bring the physics of memory to foundation models💪
Dmitry Krotov Verified account @DimaKrotov Apr 9
GPT is a standard paradigm for language models:
👉 tokens → neural net → shifted tokens

Energy-based models (Hopfield style) is a different paradigm:
👉 tokens evolve on an energy landscape

What if you could have both? In our new #ICLR2026 paper, we build a hybrid:
• Tokens evolve over time (like an energy-based model)
• Dynamics runs on a learned energy landscape
• Then converges to a steady state, which defines the next token

So instead of a single forward pass:
👉 prediction = a dynamical process

Our model, called NRGPT:
• Combines GPT-style inference with EBM dynamics
• Enables iterative refinement of token representations
• Opens the door to more stable architectures

And it’s not just theory:
✅ Strong results on ListOps
✅ Works on real text

Paper: arxiv.org/abs/2512.16762.

What would you build with dynamical tokens?
View quoted post on X