Twitxy Saved

Sebastian Loeschcke

@SLoeschcke

PhD at @DIKU_Institut | quantization, efficient training, low-rank structures.
Copenhagen sebulo.github.io Joined December 2017
612 Following 248 Followers 115 Posts

Latest posts 5

Sebastian Loeschcke @SLoeschcke
🧵 New paper: SQuAT: Spherical Lattice Vector Quantization-Aware Training

We scale lattice VQ-QAT to billion-parameter LLMs and 10B training tokens, outperforming standard VQ-QAT and integer QAT at 2.25 bits per weight.

Work from my internship at Qualcomm AI Research 👇
Sebastian Loeschcke @SLoeschcke
Check out our new preprint 𝐓𝐞𝐧𝐬𝐨𝐫𝐆𝐑𝐚𝐃.
We use a robust decomposition of the gradient tensors into low-rank + sparse parts to reduce optimizer memory for Neural Operators by up to 𝟕𝟓%, while matching the performance of Adam, even on turbulent Navier–Stokes (Re 10e5).
2 replies 4 reposts 23 likes 2.5K views View on X