Over the weekend, we released our fast endpoint for DSv4.1 Flash.
Relace is currently #1 on OpenRouter, and there's still a lot of room for improvement!
Relace is currently #1 on OpenRouter, and there's still a lot of room for improvement!
Eitan Borgnia
Verified account This account is verified. Learn more@EBorgnia
Cofounder @relace_ai
219 Following 695 Followers 258 Posts
Latest posts 5
OpenRouter is the largest open market for tokens, and has been essential in bringing healthy competition to the LLM inference space.
Inference is complicated and can be optimized on multiple axes (cost, throughput, accuracy), so as an exchange they have the difficult task of deciding how to route between providers.
This update now allows providers to distinguish themselves across any/all of these axes, and has pushed providers like us to improve our serving engines in ways that actually help the end user.
It's been fun trying to red team the algorithm as OR shipped some of the initial changes, and we've really appreciated how transparent the team has been throughout the transtion.
If you haven't read up on the new system, you should check out the docs! It's really quite beautiful.
openrouter.ai/docs/guides/pr…
Inference is complicated and can be optimized on multiple axes (cost, throughput, accuracy), so as an exchange they have the difficult task of deciding how to route between providers.
This update now allows providers to distinguish themselves across any/all of these axes, and has pushed providers like us to improve our serving engines in ways that actually help the end user.
It's been fun trying to red team the algorithm as OR shipped some of the initial changes, and we've really appreciated how transparent the team has been throughout the transtion.
If you haven't read up on the new system, you should check out the docs! It's really quite beautiful.
openrouter.ai/docs/guides/pr…
openrouter.ai How OpenRouter Routes Traffic Between Providers How OpenRouter splits traffic between provider endpoints: adjusted price, speed, the competitive edge, and the inputs that decide your share. We just shipped our biggest improvement to our provider router since the founding of OpenRouter.
Before, we prioritized mostly based on price. The new router now rewards the best quality inference for the actual cost of a request, so requests stay cheap but are also fast and reliable.
Providers that deliver more value automatically win more traffic, so every provider has a reason to keep improving. And every time a provider gets better, so does your inference on OpenRouter.View quoted post on X
Separating API requests into different tiers of SLAs is a core problem of inference at scale.
It's also incredibly difficult to price. Using a standard fixed contract requires constructing some function of minimum TPS, maximum TTFT, workload shape --> input/cached input/output token rates.
It turns out this function is non-linear in all of these input variables.
Ideally, you'd like to have a single central deployment per model that can handle multiple customer requirements simultaneously.
Great work by @sidpra5ad @ashertrockman @keegan_w_harris and @nhaghtal on solving this by blending an auction directly into the inference engine.
It's also incredibly difficult to price. Using a standard fixed contract requires constructing some function of minimum TPS, maximum TTFT, workload shape --> input/cached input/output token rates.
It turns out this function is non-linear in all of these input variables.
Ideally, you'd like to have a single central deployment per model that can handle multiple customer requirements simultaneously.
Great work by @sidpra5ad @ashertrockman @keegan_w_harris and @nhaghtal on solving this by blending an auction directly into the inference engine.
We design an auction for allocating inference priority in LLM serving systems. It is carefully designed to maximize economic efficiency while retaining key performance metrics (cache hits and mean TTFT) of the underlying SGLang inference engine.View quoted post on X
inferenceauctions.com
We're just happy to be included :)
Together is the #1 provider by volume of tokens (across closed and open endpoints) on @OpenRouter
openrouter.ai/providersView quoted post on X
I'm more and more convinced startups need a small number of chaos creators leading the charge on new projects.
Then you have an army of meticulous members of the team to take the ideas that work and make them rock solid.
The funny thing is as we've hired, I've gone from being meticulous one to being the source of chaos.
Then you have an army of meticulous members of the team to take the ideas that work and make them rock solid.
The funny thing is as we've hired, I've gone from being meticulous one to being the source of chaos.