Pinned post
AutoText's typing suggestions are now 2x faster!
We're at 800ms RTT per suggestion, still slower than I'd like. For reference, Cursor's gold-standard suggestions land around 250ms.
Inference dominates the latency budget, eating 93%+ of the round trip. After benchmarking dozens of models across every major serverless provider, I've hit a ceiling.
One lever remains: dedicated deployments running open-source models (Kimi K2, Qwen3, GLM, MiniMax, Gemma 4) on hand-picked hardware. That unlocks the accelerators @FireworksAI_HQ and @togethercompute reserve for dedicated tenants, namely speculative decoding, predicted outputs, and their fastest chips. The catch: always-on capacity (no scale-to-zero) starts at $7k/month.
We're at 800ms RTT per suggestion, still slower than I'd like. For reference, Cursor's gold-standard suggestions land around 250ms.
Inference dominates the latency budget, eating 93%+ of the round trip. After benchmarking dozens of models across every major serverless provider, I've hit a ceiling.
One lever remains: dedicated deployments running open-source models (Kimi K2, Qwen3, GLM, MiniMax, Gemma 4) on hand-picked hardware. That unlocks the accelerators @FireworksAI_HQ and @togethercompute reserve for dedicated tenants, namely speculative decoding, predicted outputs, and their fastest chips. The catch: always-on capacity (no scale-to-zero) starts at $7k/month.
0:43