Sebastian Raschka

I put together a mega write-up on GPT-6 Astra & looped transformers.

How looped transformers / recurrent depth works, cost-tradeoffs, whether it hides reasoning traces, with lots of figures and a tour of recent looped transformer research.

What could possibly go wrong?

Catching up on the 10 (!) open LLM architecture releases from the past month. Quite a lot happened in October.
I guess we can call it "Open October"!

1. Kolibri-1 (78B): A notable model from Germany. Features classic Grouped Query Attention (GQA) and Sliding Window Attention (SWA) attention module.

2. Ling 3.1 Flash (560B): Uses Kimi Delta Attention and the eighth uses gated Multi-head Latent Attention (MLA) inspired by DeepSeek.

3. Muse Spark Open (816B): Besides Musse Glimmer, the first major open weight release from Meta (since Llama). Uses classic GQA + SWA but adds a LatentMoE.

4. Nemotron 3.5 Super VL (120B): Combines Mamba-2 layers with just eight attention layers. Also LatentMoE

5. Dory1.0 (4B): A looped transformer design.

6. Reflection Beam (501B): GQA+SWA architecture.

7. Recursive Flint (40B): Note that "Recursive" is the company name; the architecture is a fairly standard MoE with GQA and SWA.

8. MiniMax M3.1-Flash (428B): Uses their own MiniMax Sparse Attention module.

9. LFM3 (1.7B): Hybrid architecture that mixes short convolutions and attention.

10. Mistral Large 4 (1T) combines MLA with sparse attention.

Will probably cover/discuss things in more depth in the coming weeks!

The complete reasoning-from-scratch video series is now up (and linked) here: magazine.sebastianraschka.com/p/building-rea…

(Taking a short break but will be back with LLM model and research updates in November!)

Reasoning From Scratch round 8, the finale!
This last lecture in the series covers distillation as an efficient alternative to the previous reinforcement learning training runs.

(Planning to take a brief break, but I will be back!)

00:00 Introduction and series recap
01:07 Why distill a reasoning model?
02:51 Soft vs. hard distillation
11:06 On-policy vs. off-policy distillation
18:09 Preparing data for reasoning distillation
22:35 Loading the DeepSeek R1 MATH dataset
28:04 Inspecting reasoning traces and final answers
31:27 Tokenization and chat templates
35:58 Pre-tokenizing the training dataset
39:01 Sequence lengths and the 2,048-token limit
42:38 Creating training and validation splits
45:32 Loading the Qwen3-0.6B base model
46:24 Computing answer-token log probabilities
54:10 Cross-entropy loss for distillation
57:25 Training and validation loss helpers
1:02:03 Implementing the distillation training loop
1:07:26 Running the standalone training script
1:09:41 Plotting training and validation losses
1:11:31 Evaluating checkpoints on MATH-500
1:14:09 Comparing teachers and distillation results
1:16:34 Future directions for reasoning models
1:19:41 Bonus exercises and next steps