vLLM

This account is verified. Learn more

@vllm_project

A high-throughput and memory-efficient inference and serving engine for LLMs. Join slack.vllm.ai to discuss together with the community!

vllm.ai Joined March 2024

36 Following 54K Followers 1.5K Posts

Latest posts 5

vLLM @vllm_project 2d

76 guest speakers, 24 organizations, 59 sessions — and every one driven by this community 📷 Thank you @RedHat_AI and all our guests. Join us bi-weekly, open to all: vllm.ai/office-hours

vllm.ai vLLM Office Hours A bi-weekly livestream with vLLM committers: project updates, guest talks, and open Q&A. Includes recordings and slides from past sessions.

Red Hat AI Verified account @RedHat_AI 2d

76 guest speakers. 24 organizations. 59 sessions and counting.

vLLM Office Hours is a bi-weekly livestream with the people building @vllm_project: a project update, a guest talk, and open Q&A. Open to all.

Get a calendar invite: vllm.ai/office-hours

View quoted post on X

vLLM @vllm_project 2d

Love seeing what the community builds on vLLM-Omni 🤩 OpenDirector: a local short-film workspace — generate, edit, export — powered by vLLM-Omni with a wide range of models. Congrats @hsliuunc! 🎬

🎥 OpenDirector: github.com/ThinkFlowLab/O…
⚡ vLLM-Omni: github.com/vllm-project/v…

github.com GitHub - ThinkFlowLab/OpenDirector: Build an easy-to-use workflow for the next generation of video... Build an easy-to-use workflow for the next generation of video creation with agent - ThinkFlowLab/OpenDirector

Hongsheng Liu Verified account @hsliuunc 2d

Introducing OpenDirector v0.1.0 powered by vLLM-Omni × MiniMax H3 🎬

Generate, edit and export short films in one local workspace. Our minutes long demo uses FastH3 served through vLLM-Omni.

Try it and share your feedback:
github.com/ThinkFlowLab/O…

#vllmomni #MiniMaxH3 @vllm_project @rogerw0108 @90VuHwwuzNhhW3h @KaichaoYou

View quoted post on X

vLLM @vllm_project 3d

One router, two minds 🧠⚡

Huge work from @XunzhuoLiu and the vLLM Semantic Router team on System Duo — fast decision models handle the blink-of-an-eye calls (triage, guard, verify, check), while the router sends deep reasoning to the right model only when it matters.

All open source, all running on vLLM → app.vllm-sr.ai

app.vllm-sr.ai System Duo · Thinking, fast and slow. One key. Fast decisions and deep reasoning from open models. One free key.

Xunzhuo @XunzhuoLiu 3d

𝗠𝗮𝗸𝗲 𝘆𝗼𝘂𝗿 𝗮𝗴𝗲𝗻𝘁 𝘁𝗵𝗶𝗻𝗸 𝗳𝗮𝘀𝘁 𝗮𝗻𝗱 𝘀𝗹𝗼𝘄.

System One decides in a blink. System Two reasons in depth.

𝗢𝗻𝗲 𝗿𝗼𝘂𝘁𝗲𝗿. 𝗧𝘄𝗼 𝗺𝗶𝗻𝗱𝘀.

𝗜𝗻𝘁𝗿𝗼𝗱𝘂𝗰𝗶𝗻𝗴 𝗦𝘆𝘀𝘁𝗲𝗺 𝗗𝘂𝗼 → app.vllm-sr.ai

#Jev #SystemOne #SystemTwo #OpenSource

View quoted post on X

vLLM @vllm_project 3d

Kimi K3.1 Pro on TPUs, with day-0 support ⚡

Run it on TPU v7x with vllm-torchtpu. Great to see the continued collaboration between the @Google and @inferact teams! 🤝

Read more 👇
llm-d.ai/blog/deploy-ki…

llm-d.ai Serving Moonshot AI's Kimi K3.1 Pro with llm-d across TPUs and GPUs | llm-d Day-0 distributed serving recipes for Moonshot AI's 2.8T-parameter Kimi K3.1 Pro at 1M context using llm-d across Ironwood TPU v7x (SGLang) and NVIDIA GB300 GPUs (vLLM), on Google Cloud.

vLLM @vllm_project 3d

Thanks to @SemiAnalysis_ for highlighting the vLLM ecosystem’s work on TPU, in particular the Kimi K3 megakernel and vLLM TorchTPU powering AgentX results across GLM-5.2, Qwen3.5 397B, and Kimi K3.

Shoutout to @inferact and @googlecloud for pushing vLLM performance on TPU. The @SemiAnalysis_ write-up below is a great read 👇

SemiAnalysis Verified account @SemiAnalysis_ 3d

The Iron(wood) Throne:
Google's TPU Stakes Its Claim on Inference
Viable Alternative of CUDA Moat,
NVIDIA Working Nights & Weekends,
Jensen’s Counter Submission, Kimi K3,
Meta Muse Spark Open, GLM5.3,
TorchTPU, SGLang, vLLM, llm-d, Ray
newsletter.semianalysis.com/p/tpu-strikes-…
View quoted post on X