Konstantin Rusch

Neehal Tumma Verified account @ntumm120 · 5d

We've spent years learning how to quantize attention. But what about the linear recurrences (e.g. Mamba-2, GDN and KDA) replacing it?

It turns out we can rewrite their computation to reduce quantization error, while preserving the full-precision function. 🤯

Introducing LinQuant: a framework for low-precision recurrent prefill. 🧵 (1/7)

View quoted post on X

Liquid AI Verified account @liquidai · Oct 27

Introducing 𝙻𝙵𝙼𝟹: our new generation of Liquid Foundation Models. With a new tri-hybrid architecture and new multimodal encoders, the LFM3 family is the ideal model suite for specialized intelligence.

> 6 model instances, across text, audio, vision, and omni modalities, generative and decision models.
> 𝙻𝙵𝙼𝟹-𝟷.𝟽𝙱: text, 256k context window.
> 𝙻𝙵𝙼𝟹-𝚅𝙻-𝟸𝙱: vision-language model with video-input support for spatial understanding
> 𝙻𝙵𝙼𝟹-𝙰𝚞𝚍𝚒𝚘-𝟸𝙱: tool calls, voice chat, reasoning in 18 languages.
> 𝙻𝙵𝙼𝟹-𝙾𝚖𝚗𝚒-𝟸𝙱: an omni model combines vision, audio, and language.
> 𝚍𝟷-𝟸𝙱 and 𝚍𝟷-𝙾𝚖𝚗𝚒-𝟸𝙱: decision models with typed outputs and calibrated probabilities in a single pass. Omni adds audio input.
> fine-tune on 1 GPU.
> a new tri-hybrid architecture
> architecture optimized for 14 processor classes across GPUs, CPUs, and NPUs.
> 𝙻𝙵𝙼𝟹-𝚅𝙻-𝟸𝙱 is a beast. It performs competitively with Gemma 4 12B (6x larger), and Gemini 3.5 Flash Lite, and GPT-5.6 Luna, with 1.9B parameters.
> 𝙻𝙵𝙼𝟹-𝟷.𝟽𝙱 is #1 on the Pareto frontier of device intelligence 𝚙𝚒𝚙𝚎𝚝𝚝𝚎.𝚕𝚒𝚚𝚞𝚒𝚍.𝚊𝚒

Every modality for specialized intelligence. Pick the one that fits your use case, customize it, and run it on-device.

1/ 🧵

View quoted post on X

alphaXiv Verified account @askalphaxiv · Oct 20

"Depth Memory"

Attention Residuals stores earlier Transformer layer outputs for retrieval but this memory grows with depth.

Linear Depth Memory (LDM) maintains a fixed-size memory across layers, using recursive least squares to store and retrieve earlier layer information alongside the standard residual stream.

On 400M and 1.5B models, LDM achieves lower perplexity than standard residuals, mHC, and AttnRes, while maintaining competitive zero-shot accuracy and comparable throughput with optimized kernels.

alphaxiv.org/abs/2610.18809

View quoted post on X

Sumin Park @psmiziz · Oct 15

Can reasoning finish while latent states keep moving?

Happy to share Quotient Recurrent Reasoner (QRR)!

We let symmetry-generated rotations persist while task-relevant dynamics converge faster in quotient space, reaching 97.9% on Sudoku-Extreme with 78% fewer loops than FPRM.

View quoted post on X

Sumin Park @psmiziz · Oct 15

Can reasoning finish while latent states keep moving?

Happy to share Quotient Recurrent Reasoner (QRR)!

We let symmetry-generated rotations persist while task-relevant dynamics converge faster in quotient space, reaching 97.9% on Sudoku-Extreme with 78% fewer loops than FPRM.

View quoted post on X