Pinned post
Today we are releasing our speculative decoding implementation in our inference engine uzu.
Initially for Qwen3.6 27B, with support for Qwen3.8 27B and Muse Glimmer coming soon.
On Apple M5-series chips, we outperform MTPLX (MLX + speculative decoding) by almost 2x, and llama.cpp by over 3x at comparable quantization levels, with the strongest gains achieved on mathematical reasoning and coding tasks.
Run the model:
Mirai-M: trymirai.com/local-models/a…
Mirai-L: trymirai.com/local-models/a…
Explore the benchmarks:
trymirai.com/metrics
Learn more about our speculative decoding implementation:
trymirai.com/blog/speculati…
Our draft model, quantized checkpoint format, verification algorithm, and GPU kernels are co-designed from the ground up around the latest Apple M5 chips to take maximum advantage of GPU Neural Accelerators.
Unlike popular speculative decoding architectures such as model-native MTP, which produce small draft chains of 3-4 tokens at a time, we use extremely aggressive speculative budgets of 16-32 tokens. This enables us to use Neural Accelerator-backed GEMM kernels, achieving maximum utilization of hardware arithmetic throughput.
Initially for Qwen3.6 27B, with support for Qwen3.8 27B and Muse Glimmer coming soon.
On Apple M5-series chips, we outperform MTPLX (MLX + speculative decoding) by almost 2x, and llama.cpp by over 3x at comparable quantization levels, with the strongest gains achieved on mathematical reasoning and coding tasks.
Run the model:
Mirai-M: trymirai.com/local-models/a…
Mirai-L: trymirai.com/local-models/a…
Explore the benchmarks:
trymirai.com/metrics
Learn more about our speculative decoding implementation:
trymirai.com/blog/speculati…
Our draft model, quantized checkpoint format, verification algorithm, and GPU kernels are co-designed from the ground up around the latest Apple M5 chips to take maximum advantage of GPU Neural Accelerators.
Unlike popular speculative decoding architectures such as model-native MTP, which produce small draft chains of 3-4 tokens at a time, we use extremely aggressive speculative budgets of 16-32 tokens. This enables us to use Neural Accelerator-backed GEMM kernels, achieving maximum utilization of hardware arithmetic throughput.
