Twitxy Saved

Yichuan Wang

Verified account This account is verified. Learn more

@YichuanM

phd candidate @ uc berkeley (skylab, bair).
mlsys.
creator of leann: github.com/yichuan-w/leann
sjtu acm class alum.
yichuan-w.github.io Joined May 2021
2.5K Following 2.9K Followers 646 Posts

Latest posts 5

Pinned post

Yichuan Wang Verified account @YichuanM
Zero Trainโ€“Inference Mismatch โ€” now for linear attention, and under async RL ๐ŸŽฏ

We got bitwise-exact trainer/generator parity for Gated DeltaNet (Qwen3.5-9B / 35B-A3B) on TorchTitan RL + vLLM, then asked the question nobody had actually tested in open source: does it help async RL?

Two firsts (as far as we know):
1๏ธโƒฃ First open-source stack to hit train/inference logprob diff = 0 on a
linear-attention model
2๏ธโƒฃ First to measure what zero mismatch is worth under async off-policy RL

How โ€” smaller than it sounds:
โ€ข trainer and generator share one model definition (TorchTitan unified model). No more aligning trainer & generator op by op
โ€ข recurrent kernel everywhere on the forward (prefill + decode + trainer), chunked only for backward
โ€ข split-K off, batch-invariant GEMM, + a batch-invariant bmm for the MoE router
โ€ข we never touch FLA kernel internals โ€” the recurrent kernel is intrinsically batch-invariant: one sequence's own state, fixed order, no cross-sequence reductions
โ€ข vLLM's linear-cache management + prefix caching keep working as-is

๐Ÿ“Š Qwen3.5-9B + DAPO-Math, async at offpolicy = 4 / 12 / 32:
โ€ข logprob_diff = exactly 0 at step 0 โœ… every nonzero after that is staleness, not precision
โ€ข at offpolicy=32 the stock stack runs away past 0.065; ours holds ~0.035
โ€ข the unified model is free โ€” the entire bill is the kernels
โ€ข also verified on MoE 35B-A3B and a 64-turn / 64K-context terminal agent

๐Ÿค” Now the honest part: it depends on the config. On math at offpolicy=12, BI leads on train reward and peaks highest on held-out AIME. On the terminal agent it's slightly ahead. On search, and on other math configs, the curves are indistinguishable. What is consistent is the numerics โ€” BI keeps the gap from exploding as you widen the window. Cost: 2โ€“3ร— trainer throughput (โ‰ˆ5ร— on the agent).

Our take: bitwise parity is a debugging tool, not a production default. Flip BI on for 20 on-policy steps โ€” if the gap is 0 and your run is still broken, the infra is exonerated and the bug is in your data or your algorithm. That's worth 20 steps, not a whole run.

Next:
โ€ข make BI cheap (kill the 2โ€“3ร—)
โ€ข more workloads, more seeds
โ€ข stay tuned for more complex agentic training recipes/tests ๐Ÿ‘€
Full writeup ๐Ÿ”— yichuan-w.github.io/blog/GDN-trainโ€ฆ

We collaborated closely with the TorchTitan team, and we sincerely encourage everyone to check out TorchTitan RL. With its clean, efficient unified model architecture and native async RL design, we believe it is a strong foundation for your next RL engine.

#LLM #RL #agentic #pytorch #torchtitan
28 replies 92 reposts 537 likes 133K views View on X
Yichuan Wang Verified account @YichuanM
Thereโ€™s still so much to explore in this field, from research to real-world products, and so much potential to build things that truly help people.

I think deeply personal agents may be harder for companies like OpenAI and Anthropic (any frontier lab) to pursue, given the privacy concerns around sensitive user data. That leaves a lot of room for personal computer-use agents and local agents built around our own indexes (for all startup and research groups).

Iโ€™m starting to collect my own data, too, with the goal of eventually training an agent tailored to me (as I am more into agentic RL recently).

Whether itโ€™s SFT data, RL environments, all these deep-research-style tasks, knowing the outcome makes it easier to work backward and synthesize tasks and training examples grounded in our own activity.

The most exciting part? Only you have your full personal context. You can build an agent thatโ€™s uniquely yoursโ€”tailored to how you think and work, with your data staying private and under your control.
Yichuan Wang Verified account @YichuanM 3d
Iโ€™m beyond excited about this. When I first started working on on-device personal agents, this was the dreamโ€”and now itโ€™s real.

We built a real-time visual indexing method and brought it to life in a working product. The idea is wild: you shouldnโ€™t have to remember every little thing youโ€™ve seen or done on your computer. Your agent should remember it for you. ๐Ÿคฏ

Itโ€™s already helped me dig up an old Golden State Warriors ticket and find the GEN vs. BLG recap video I was watching over lunch yesterday. Those โ€œwait, where did I see that?โ€ moments? Just ask your agent.

Multimodal. Multilingual. Real-time. All running quietly on your computer, private, lightweight, and ready when you need it.

Incredible work led by @dayoon12161, and an amazing collaboration as always with @sewon__min ๐Ÿš€

๐Ÿ’ป Code github.com/StarTrail-org/โ€ฆ
๐Ÿ“„ Paper arxiv.org/abs/2611.18516
View quoted post on X
Yichuan Wang Verified account @YichuanM
Iโ€™m beyond excited about this. When I first started working on on-device personal agents, this was the dreamโ€”and now itโ€™s real.

We built a real-time visual indexing method and brought it to life in a working product. The idea is wild: you shouldnโ€™t have to remember every little thing youโ€™ve seen or done on your computer. Your agent should remember it for you. ๐Ÿคฏ

Itโ€™s already helped me dig up an old Golden State Warriors ticket and find the GEN vs. BLG recap video I was watching over lunch yesterday. Those โ€œwait, where did I see that?โ€ moments? Just ask your agent.

Multimodal. Multilingual. Real-time. All running quietly on your computer, private, lightweight, and ready when you need it.

Incredible work led by @dayoon12161, and an amazing collaboration as always with @sewon__min ๐Ÿš€

๐Ÿ’ป Code github.com/StarTrail-org/โ€ฆ
๐Ÿ“„ Paper arxiv.org/abs/2611.18516
1:22
Dayoon Ko @dayoon12161 3d
Wait, what was that paper I saw last week with the pink and blue Pareto curves? ๐Ÿค” Keep screenshots of your screen, and you can find it again, right on your laptop. ๐Ÿ’ป

๐Ÿ”ฅ Introducing ๐‘บ๐’•๐’“๐’†๐’‚๐’Ž๐‘ฌ๐’Ž๐’ƒ๐’†๐’…: real-time visual indexing of your screen history, fully on-device.

โš–๏ธ The latencyโ€“accuracy trade-off
OCR text index: fast, but loses what only pixels show
Screenshot index: accurate, but 2.3s per screenshot on a laptop GPU

๐‘บ๐’•๐’“๐’†๐’‚๐’Ž๐‘ฌ๐’Ž๐’ƒ๐’†๐’… gets both: ๐˜‚๐—ฝ ๐˜๐—ผ ๐Ÿฏ.๐Ÿณร— ๐—ณ๐—ฎ๐˜€๐˜๐—ฒ๐—ฟ than full-screenshot encoding at matched accuracy, and +๐Ÿฒ.๐Ÿฏ ๐—ฅ๐—ฒ๐—ฐ๐—ฎ๐—น๐—น@๐Ÿญ๐Ÿฌ over OCR.

๐Ÿ’ซ How?
(1) ๐—ฆ๐—ฐ๐—ฟ๐—ฒ๐—ฒ๐—ป๐˜€ ๐—ฏ๐—ฎ๐—ฟ๐—ฒ๐—น๐˜† ๐—ฐ๐—ต๐—ฎ๐—ป๐—ด๐—ฒ. Often, only a handful of blocks differ.
(2) ๐—ฆ๐—ผ ๐˜„๐—ฒ ๐˜€๐˜๐—ผ๐—ฝ ๐—ฟ๐—ฒ๐—ฐ๐—ผ๐—บ๐—ฝ๐˜‚๐˜๐—ถ๐—ป๐—ด ๐˜๐—ต๐—ฒ๐—บ. Spatial token pruning drops blocks that repeat a neighbor; temporal KV reuse recomputes only what changed: 57โ€“71% less compute.
(3) ๐—” ๐˜๐—ถ๐—ป๐˜† ๐—ฎ๐—ฑ๐—ฎ๐—ฝ๐˜๐—ฒ๐—ฟ keeps the embedding faithful to the full screenshot.

๐Ÿ“ ๐—ง๐—ฎ๐—ธ๐—ฒ๐—ฎ๐˜„๐—ฎ๐˜†: keeping your entire screen history as pixels is now practical on a laptop. Next: fully local visual agents with memory.

You can try it on your laptop right now: ๐™ฅ๐™ž๐™ฅ ๐™ž๐™ฃ๐™จ๐™ฉ๐™–๐™ก๐™ก ๐™จ๐™ฉ๐™ง๐™š๐™–๐™ข๐™š๐™ข๐™—๐™š๐™™
๐Ÿ’ป Code: github.com/StarTrail-org/โ€ฆ
๐Ÿ“„ Paper: arxiv.org/abs/2611.18516
View quoted post on X
6 replies 7 reposts 64 likes 138K views View on X
Yichuan Wang Verified account @YichuanM
Beyond Score Centering: what else can we fix in async RL? ๐Ÿงต

Score centering (SC) is the hot new trick for off-policy LLM RL. We stress-tested it under real async staleness (every batch up to 127 steps old) and found what it fixes, what it doesn't, and two cheap wins on top

๐Ÿ” What SC actually fixes
โ€ข Without SC, a trajectory's gradient norm grows ~linearly with its age: at age 64 it's 3.2ร— an on-policy one. With SC: 1.5ร—
โ€ข The culprit is a hidden drift term, an uncontrolled โˆ’AยทKL(ฮผโ€–ฯ€) baked into the policy gradient. SC removes it

โš ๏ธ What SC + IS gets wrong
โ€ข SC then IS โ†’ adds a new drift
โ€ข IS then SC โ†’ a no-op without truncation; with truncation it centers on a distribution with no guarantee of being closer to ฯ€
โ€ข So "IS and SC are orthogonal" (from the SC paper) doesn't hold

Two wins on top of SC
1๏ธโƒฃ State-level KL mask: drop states where KL(ฮผโ€–ฯ€) > ฯ„, instead of masking by token-level ratios. It depends only on the state, so SC's baseline property survives. At age 127: 3/3 seeds climb steadily, while TIS+SC and MIS+SC spike and crash
2๏ธโƒฃ Random buffer sampling, a free lunch from the system: same buffer, sample at random instead of FIFO. Mean age 127 โ†’ 52, better for every method at equal wall-clock

Recipe = SC + KL penalty + KL mask. At lr 4e-2 (w/ SGD), MIS+SC and DPPO collapse and TIS+SC swings, while every recipe seed trains and learns faster early

๐Ÿค” So far, weโ€™ve only tested on Countdown with Qwen3-0.6B (wants to scale it up!). When staleness is low, all the fixes perform similarly, but the differences become clearer as staleness increases. We also havenโ€™t compared against sequence-level masking (which might already be enough!) or isolated kernel mismatch from async lag. Both are worth looking into next.

#LLM #RL #AsyncRL #ReinforcementLearning
23 replies 42 reposts 351 likes 309K views View on X
Yichuan Wang Verified account @YichuanM
Nice empirical results! We need a solid study of SC + async RL. Lots still open:
โ€ข SC alone โ‰  on-policy gradient in expectation (can even flip sign)
โ€ข Full trajectory-level IS is unbiased, but variance explodes with length
โ€ข Truncation โ†’ SC+TIS keeps residual bias under lag

Need more principled, scientific work on async RL for LLMs, esp. long-horizon tasks.
Shu Lynn Liu Verified account @shulynnliu Nov 5
๐Ÿค” Is score centering (SC) all we need for stable async RL?

People often stack truncated importance sampling (TIS) or KL penalties on top of SC. In a fun side project, we tested how much they help.

We found that SC alone often matched on-policy. But when it fell short, adding more corrections doesn't necessarily help. ๐Ÿงต
View quoted post on X