Twitxy Saved

Sean McLeish

Verified account This account is verified. Learn more

@SeanMcleish

PhD student at the University of Maryland
mcleish7.github.io Joined November 2023
171 Following 722 Followers 231 Posts

Latest posts 5

Pinned post

Sean McLeish Verified account @SeanMcleish
Library of Layers: models which adapt layer order per input, routing over whole transformer blocks.

This opens up a new avenue to control: the execution path the model uses. We use this to control the algorithm used to solve mazes and sudoku by only changing the layer order.
🧵 1/n
9 replies 33 reposts 133 likes 7.4K views View on X
Sean McLeish Verified account @SeanMcleish
Loss spikes appear to be coming from the growing lm head norm. We trace it further, finding a growing lm head norm causes increased quantisation error and a snowball effect that culminates in an irreversible loss spike.
The fix: regularise to stay where the finite precision is high
Micah Goldblum Verified account @micahgoldblum Nov 3
LLM training runs can look healthy for billions of tokens or more then suddenly diverge. We find that quantization error in the LM head drives these delayed time-bomb loss spikes. A simple optimizer change can prevent them. 🧵 1/n
View quoted post on X
2 replies 7 reposts 70 likes 4.7K views View on X
Sean McLeish Verified account @SeanMcleish
Humans don’t maintain exact, line-by-line recall of huge contexts like full codebases or long legal documents. We keep a high-level mental model, then look things up when precision matters. We enable LLMs to do this, with high speed.
Pavel Izmailov Verified account @Pavel_Izmailov Jun 10
New paper: Latent Context Language Models (LCLMs)!

Idea: encode 16 tokens as 1 latent token, and have the LLM work on top of the latent tokens. Result: general-purpose model with much better performance / speed / memory usage frontier.
View quoted post on X
0 replies 4 reposts 21 likes 2.9K views View on X
Sean McLeish Verified account @SeanMcleish
Scaling context works but it is expensive. Prior KV cache compression promises answers but never fully delivers vLLM speed, we solve this.
Leon Verified account @iamleonli Jun 9
How far can we compress the discrete tokens in an LLM's context into compact latent vectors?

With the right training recipe at large scale, our Latent Context Language Models (LCLMs) compress context up to 16× and land on a new Pareto frontier for long-context inference. 🧵(1/n)
View quoted post on X
0 replies 2 reposts 6 likes 1.3K views View on X
Sean McLeish Verified account @SeanMcleish
Offline recurrence can improve inference accuracy by iteratively refining fast weights, giving the model an adaptation mechanism at test time.
Sangyun Lee @sang_yun_lee May 28
Almost all animals sleep. Why don’t LMs?

Introducing our new work on language model sleep.

tl;dr : A periodic, recurrent “sleep” phase allows LMs to digest their context and transfer it into their weights, improving recall and reasoning on challenging tasks.
View quoted post on X