🎉Exciting joint work between Amazon #SafeMaker and #LMCache!
Prefill and decode want opposite things from a GPU. Chunked prefill reduces the interference without removing it.View quoted post on X
On Amazon SageMaker HyperPod the two phases get separate GPU pools, and LMCache's PD backend moves the KV cache between them. Per-token latency then stops climbing with concurrency, where a colocated deployment degrades.
Whether that pays off comes down to one question: can the KV cache move between pools faster than a decoder can recompute it?
Xuan Lu, Vinay Arora, and Swapnil Palod at AWS, with Yihua Cheng at Tensormesh, measured it on Llama-3.3-70B-Instruct and wrote up the NIXL and EFA path that makes the answer yes.
blog.lmcache.ai/en/2026/09/15/…