Twitxy Saved

Junchen Jiang

Verified account This account is verified. Learn more

@JunchenJiang

Chicago, IL Joined September 2012
324 Following 1.1K Followers 194 Posts

Latest posts 5

Junchen Jiang Verified account @JunchenJiang
🎉Exciting joint work between Amazon #SafeMaker and #LMCache!
LMCache Lab Verified account @lmcache Sep 15
Prefill and decode want opposite things from a GPU. Chunked prefill reduces the interference without removing it.

On Amazon SageMaker HyperPod the two phases get separate GPU pools, and LMCache's PD backend moves the KV cache between them. Per-token latency then stops climbing with concurrency, where a colocated deployment degrades.

Whether that pays off comes down to one question: can the KV cache move between pools faster than a decoder can recompute it?

Xuan Lu, Vinay Arora, and Swapnil Palod at AWS, with Yihua Cheng at Tensormesh, measured it on Llama-3.3-70B-Instruct and wrote up the NIXL and EFA path that makes the answer yes.

blog.lmcache.ai/en/2026/09/15/…
View quoted post on X
0 replies 1 reposts 10 likes 983 views View on X
Junchen Jiang Verified account @JunchenJiang
🚨New paper alert!

Context: KV cache has a boom in research -- 40x more papers on it in 2026 than 2024. But our experience shows that the industry can be reluctant because they are all *LOSSY* --- meaning the results with KV cache compression can be different from without it.

VeriCache solves exactly that problem without sacrificing the performance gain from the KV cache sparsification techniques.

Great work by the team led by Jiayi Yao (@EricYao43125144) !
Jiayi Yao Verified account @EricYao43125144 Jul 1
View quoted post on X
1 replies 6 reposts 24 likes 2.7K views View on X
Junchen Jiang Verified account @JunchenJiang
I was delighted to give three keynotes recently, one at the 3rd HotInfra workshop at ISCA'26 (hosted by Prof. Jian Huang), one at the UIUC Systems Groups retreat (hosted by @tianyin_xu ), and a CMU guest lecture (hosted by Prof. Lei Li).

The message was simple: KV cache is BEYOND cacheable intermediate states.

It is what makes inference systems SMART.

- If cached, it saves cost and reduces latency.
- If analyzed, it reveals how the model attends to and understands context.
- If optimized, it can improve not only performance, but also output quality.

It's not a liability, but an ASSET --- (almost) always the more you have, the better.

That is the part most people in industry still underestimate, even though the techniques are already here today: KV cache compression, blending, semantic reuse, cross-model sharing, attention steering, KV-cache-based retrieval, and more.

The gap is not imagination. The gap is productization.

KV cache will become a core AI-native data layer in future inference infrastructure — not just something we store, but something we manage, analyze, and optimize.

#KVCache, #LMCache, #LLMInference, #Tensormesh, #vLLM, #SGLang, #MLSys, #HotInfra