We design an auction for allocating inference priority in LLM serving systems. It is carefully designed to maximize economic efficiency while retaining key performance metrics (cache hits and mean TTFT) of the underlying SGLang inference engine.
inferenceauctions.com
inferenceauctions.com
When LLM demand exceeds capacity, which requests should go first?
Model providers already offer a few priority tiers. But even within a tier, requests compete for compute, and users value latency differently. Your coding agent might be blocked due to a request that belongs to an unattended background job.
Why not let the market decide?
In our new paper, Inference Auctions, users bid for request priority, and our auction allocates that priority to those who value it most.
Unfortunately, naively selling to the highest bidders destroys prefix-cache locality, increasing SGLang's average latency by up to 12x in our experiments.
Our fix is to run an auction over depth-first search traversals of the request prefix tree, ordering subtrees by average bid. This maximizes social welfare while retaining cache optimality, with VCG payments incentivizing truthful bidding, and the resulting request order preserves key performance metrics (cache hits and mean TTFT) of the underlying scheduler.
w/ @keegan_w_harris @sidpra5ad @nhaghtal and Michael Jordan
Paper: arxiv.org/abs/2609.40070
Project page: inferenceauctions.comView quoted post on X
