Awesome FlashState lead by Weili!! A high prefix cache hit rate is critical for efficient real-time agents in the wild. In streaming video interaction, new frames and user queries are appended to the end of the context, and earlier frames stay unchanged. Looking forward to seeing related work on video!
Introducing FlashState âš¡: an inference system that flexibly manages the prefix cache of hybrid LLMs, designed for agentic workloads.
On GLM 5.3 Flash, FlashState more than doubles vLLM's throughput and provider profit, while cutting latency by 3.7× and user cost by 2.5×.View quoted post on X
