Twitxy Saved

LMCache Lab

Verified account This account is verified. Learn more

@lmcache

🧪 Open Source Team that maintains LMCache and Production Stack
🤖 Democratizing AI by providing efficient LLM serving for ALL
Github, Online lmcache.ai Joined September 2024
59 Following 2.2K Followers 319 Posts

Latest posts 5

LMCache Lab Verified account @lmcache
✨ LMCache v0.5.7.post1 is now available, with safer defaults, more accelerators, and native gRPC.

The official Docker images now run as the non-root user lmcache (UID 1000). The MP HTTP server, the frontend HTTP API, and the coordinator gain optional API key authentication. It stays off until you configure a key with --api-key-file.

Biren SUPA and Cambricon MLU are new platforms, and the gRPC MP transport adds native and asyncio server modes. Packages are built against PyTorch 2.14.0.

This release includes work from 34 new contributors. Thank you, and welcome.

âš™ Before you upgrade: make sure UID 1000 can read and write every path LMCache uses, including mounted volumes.

na2.hubs.ly/H093XcG0

#LMCache #KVCache
blog.lmcache.ai LMCache v0.5.7.post1: Safer Defaults, New Accelerators, and Native gRPC LMCache v0.5.7.post1 runs the official images as a non-root user, adds optional API key authentication for HTTP services, and widens accelerator support.
LMCache Lab Verified account @lmcache
No GPU? You can still develop LMCache.

Our June starter guide shows a new contributor how to run vLLM and LMCache end to end on an Apple Silicon MacBook: a CPU build of vLLM, LMCache installed with NO_GPU_EXT=1, and a two-request test that shows the second request reusing the cache.

It also names four areas you can work on from a laptop: frontend, L1 eviction, L2 storage, and observability.

na2.hubs.ly/H08Slzm0

#ThrowbackThursday #Evergreen #Knowledge
blog.lmcache.ai vLLM + LMCache: A Starter Guide, No GPU Required Get started easily: a single MacBook is all you need to develop vLLM + LMCacheFor New Contributors · Covering Frontend / L1 Eviction / L2 Storage / Observability If you ever skipped LMCache because…
0 replies 1 reposts 17 likes 867 views View on X
LMCache Lab Verified account @lmcache
When a vLLM worker on an Ascend NPU restarts, does it have to lose its warm KV cache?

With LMCache multi-process (MP) mode, now supported on Ascend NPUs, the cache stays. A standalone lmcache server owns the KV cache, so the cache survives engine restarts, and several vLLM instances share it.

The harder question is how KV tensors cross the process boundary on an NPU. The new post explains the two native transfer paths, why DeepSeek-V4 works with no extra configuration, and how to start in three steps.

✅ Supported on Atlas 800I A2 and A3 and validated with Qwen models and DeepSeek-V4. The Ascend C kernels are open source.

na2.hubs.ly/H091L0N0

#LMCache #KVCache
LMCache Lab Verified account @lmcache
Last December we traced every LLM call Claude Code made while it fixed one small Django bug from SWE-bench Verified. The result: 92 calls and a 92 percent prefix reuse rate.

The post shows where that reuse comes from, including subagents and "warm-up" calls that prime the cache before the real work starts. It is a useful read if you serve agent traffic and want to know what your KV cache will see.

na2.hubs.ly/H08SlyQ0

#ThrowbackThursday #Evergreen #Knowledge
blog.lmcache.ai Context Engineering & Reuse Pattern Under the Hood of Claude Code Over the last few months, Claude Code has quietly become one of the most interesting & widely-adopted real-world agentic systems available to normal developers. Unlike cloud-only agents whose i…
8 replies 4 reposts 36 likes 959 views View on X
LMCache Lab Verified account @lmcache
When an RL job spreads sixteen samples of one prompt over four nodes, why does each node run prefill for that same prompt again?

In LMCache multi-process (MP) mode, each node’s KV cache server sees only itself. Node B does not know that node A already holds the KV.

✨ LMCache contributor Rui Zhang introduces the LMCache coordinator: a control plane that builds a fleet-wide view of every server’s cache and lets pluggable policies act on it. It is opt-in, in LMCache v0.5.6 and later, and stays out of the request path.

How it stays correct through restarts, how to add your own policy, and where it goes next:

blog.lmcache.ai/en/2026/11/11/…

#LMCache #KVCache
blog.lmcache.ai The LMCache Coordinator: A Control Plane for Fleet-Wide KV Cache The LMCache coordinator builds one fleet-wide view of KV cache across MP servers and lets pluggable policies evict, pin, and prefetch, without sitting in the request path.