Neehal Tumma

This account is verified. Learn more

@ntumm120

PhD @MIT researching efficient and continual ML | @LiquidAI | Prev CS/Math @Harvard

Cambridge, MA ntumm120.github.io Joined July 2023

298 Following 204 Followers 47 Posts

Latest posts 5

Pinned post

Neehal Tumma @ntumm120 5d

We've spent years learning how to quantize attention. But what about the linear recurrences (e.g. Mamba-2, GDN and KDA) replacing it?

It turns out we can rewrite their computation to reduce quantization error, while preserving the full-precision function. 🤯

Introducing LinQuant: a framework for low-precision recurrent prefill. 🧵 (1/7)

0:14

Neehal Tumma @ntumm120 Oct 14

Super interesting work applying test-time regression along the depth axis

David González-Martínez Verified account @_davidgonmar Oct 14

mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔

Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.

Linear Depth Memory (LDM) equips models with a matrix-valued memory.

View quoted post on X

Neehal Tumma @ntumm120 Aug 20

PGDN and PKDA have been merged into FLA, check them out! github.com/fla-org/flash-…

Neehal Tumma Verified account @ntumm120 Apr 27

Some say Gated DeltaNet > Mamba-2. Others say Mamba-2 > Gated DeltaNet.

But what if Gated DeltaNet = Mamba-2? 👀

Well maybe not exactly — but with least-squares preconditioning, we show that they reduce to the same recurrence!

We use this lens to design PDN, PGDN, and PKDA: preconditioned delta-style recurrences that outperform their unpreconditioned counterparts at scale 📈

📄 Paper: arxiv.org/abs/2604.21100 w/ @loo_noel @liquidai
💻 Code: github.com/ntumm120/preco…
View quoted post on X

Neehal Tumma @ntumm120 Jul 6

@loo_noel and I will be presenting Preconditioned DeltaNet tomorrow at ICML! Stop by if you wanna chat about recurrences, test-time training or LeBron’s free agency :)

Poster session: Hall A #2915 at 10:30am, July 7th

Neehal Tumma Verified account @ntumm120 Apr 27

Some say Gated DeltaNet > Mamba-2. Others say Mamba-2 > Gated DeltaNet.

But what if Gated DeltaNet = Mamba-2? 👀

Well maybe not exactly — but with least-squares preconditioning, we show that they reduce to the same recurrence!

We use this lens to design PDN, PGDN, and PKDA: preconditioned delta-style recurrences that outperform their unpreconditioned counterparts at scale 📈

📄 Paper: arxiv.org/abs/2604.21100 w/ @loo_noel @liquidai
💻 Code: github.com/ntumm120/preco…
View quoted post on X

Neehal Tumma @ntumm120 Jun 4

Cool paper showing the utility of data-dependent short convolutions.

One comparison I’d love to see: explicit filter conditioning used here vs the LFM style approach from @liquidai, where the short conv kernel is static but the local operator remains input-dependent via gates.

Basically dynamic vs gated static FIR filters.

Rosinality @rosinality Jun 3

arxiv.org/abs/2606.03825

Dynamic convolution on QKV! An old idea came back again.

View quoted post on X