Kimi just found a way to make AI models better at reasoning by changing how their layers talk to each other.
A language model processes your prompt one layer at a time. Normally, each layer adds its work to a running total. By the time the model reaches the deeper layers, a useful detail from earlier can get buried.
Kimi’s fix is called Attention Residuals. It lets each layer look back and decide which earlier outputs deserve more weight for the prompt it’s working on. Grouping layers into blocks keeps the memory cost down.
Here’s what changed in the Kimi Linear tests.
- Science reasoning rose from 36.9 to 44.4
- Math rose from 53.5 to 57.1
- Coding rose from 59.1 to 62.2.
In a separate experiment, the standard design needed 25% more training compute to match the new design’s prediction error.
A language model processes your prompt one layer at a time. Normally, each layer adds its work to a running total. By the time the model reaches the deeper layers, a useful detail from earlier can get buried.
Kimi’s fix is called Attention Residuals. It lets each layer look back and decide which earlier outputs deserve more weight for the prompt it’s working on. Grouping layers into blocks keeps the memory cost down.
Here’s what changed in the Kimi Linear tests.
- Science reasoning rose from 36.9 to 44.4
- Math rose from 53.5 to 57.1
- Coding rose from 59.1 to 62.2.
In a separate experiment, the standard design needed 25% more training compute to match the new design’s prediction error.