Twitxy Saved

Leon

Verified account This account is verified. Learn more

@iamleonli

PhD @nyuniversity | Researcher @modal
Prev @Columbia
leonlixyz.github.io Joined July 2024
545 Following 342 Followers 118 Posts

Latest posts 5

Pinned post

Leon Verified account @iamleonli
Scaling laws have transformed how we understand pretraining, and recent work has begun to characterize scaling in RL. But these stages are almost always studied in isolation. Can we study pretraining and RL jointly, and derive a unified scaling law for the full pipeline? 馃У
3 replies 46 reposts 294 likes 28K views View on X
Leon Verified account @iamleonli
As someone who's bitter lesson pilled, I've always been fascinated by the idea of learning model arch through gradient descent rather than relying on inductive biases.

MoE brought sparsity and dynamic routing to FFNs within individual transformer layers. Can we do the same across layers themselves?

Congrats to @arahmani_AR and @SeanMcleish on this great work!
Sean McLeish Verified account @SeanMcleish Nov 3
Library of Layers: models which adapt layer order per input, routing over whole transformer blocks.

This opens up a new avenue to control: the execution path the model uses. We use this to control the algorithm used to solve mazes and sudoku by only changing the layer order.
馃У 1/n
View quoted post on X
1 replies 4 reposts 17 likes 968 views View on X
Leon Verified account @iamleonli
Most AI benchmarks still don鈥檛 capture truly long-horizon agentic tasks. CEO bench directly fills this gap.
Tony Chen Verified account @tonychenxyz Jun 18
Today, agents execute isolated tasks. Tomorrow, agents will steer complex decisions across long horizons.

Introducing CEO-Bench, a first step to measure "Steering Intelligence." In CEO-Bench, agents are asked to run a simulated startup for 500 days.

ceobench.com
View quoted post on X
0 replies 0 reposts 3 likes 1.1K views View on X
Leon Verified account @iamleonli
How far can we compress the discrete tokens in an LLM's context into compact latent vectors?

With the right training recipe at large scale, our Latent Context Language Models (LCLMs) compress context up to 16脳 and land on a new Pareto frontier for long-context inference. 馃У(1/n)