Pinned post
EXL3 has one of the best ratios of accuracy per bit and now its also one of the fastest format for M5 and NAX enabled Macs! TurboEXL3 is a fused kernel library tested on Qwen 3.8 Flash. It can running the model up to 140TPS decode using full EXL3 conversion by @turboderp_ with equal prefill performance.
My TurboEXL3 library is faster than the standard Q4 block based quants. It achieves this through fusing the encoding and decoding kernels directly into each batched GEMM and GEMV calculation. The accuracy gains from EXL3 then boost the token drafting which offsets any performance losses from EXL3.
You can choose EXL3 at 5.05 bits for nearly lossless quality compared to BF16. Choose EXL3 @ 3.05 bits for a version with the accuracy of NVFP4 with 33% smaller size.
Available soon on mlx-serve. See the attached article for full technical details. The benchmarks were run with thinking low and greedy drafting.
My TurboEXL3 library is faster than the standard Q4 block based quants. It achieves this through fusing the encoding and decoding kernels directly into each batched GEMM and GEMV calculation. The accuracy gains from EXL3 then boost the token drafting which offsets any performance losses from EXL3.
You can choose EXL3 at 5.05 bits for nearly lossless quality compared to BF16. Choose EXL3 @ 3.05 bits for a version with the accuracy of NVFP4 with 33% smaller size.
Available soon on mlx-serve. See the attached article for full technical details. The benchmarks were run with thinking low and greedy drafting.
1:00
