@DZhang50
Pinned post
Are some parts of your model useless? 🤨
I looked impact of value residuals (cf. quoted) in Ripple 65B. I wanted to know if they had an impact in the modeling, or were just small contributions,
So I plotted the coefficient of the value residual for each layer: higher coeff = more residual is used = local hidden states matter less.
If all coefficients were 0, values residual would be useless 🥸 it is not.
The early layers use the residual more than a little, which is surprising since they are so close to the start of the model. Layer 6 even uses ONLY the residual, ie. it builds its value projection only from the input embeddings. Strength of residual peaks at layer 6 and then decreases.
Mid-layers use little to no residual: they only base their values on their own hidden states. Pretty fun.
Late layers use the residual the most: no surprise there IMO, the input signal is weakest near the end so the model benefits from a strong residual there. But it's not uniform (strong residuals are followed by weaker ones) so it looks like the model alternates between interpreting the local hidden states and looking back at the inputs.
Always cool to look into those details when a model is published! And thanks to open-source and tools like transformers, it's really easy to do. You can see the impact of an architectural decision and how it manifests in the modeling.
Fun data is just out there, waiting to be analyzed 🔎View quoted post on X
⚙️ TESTED LOCALLY: Ripple-65B Preview, a 65B open-weight model from @RicursiveAI, on my 64GB M5 Pro Mac mini
It's dense, Apache 2.0 and pretrained from scratch. I ran the community MLX 4-bit build.
- Steady 7 tok/s from 512 to 32K tokens
- Peaks at 40.5 GB, room to spare on 64GB
- 53 for 53 on my local tests, hidden facts out to 64K inc
- Runs about as fast as this chip allows for a dense 65B
The community had an MLX build up within a day.
Good day for owning your intelligence.View quoted post on X
18 of the 20 best open models today are Mixture-of-Experts. Dense is basically over.
And yet the know-how to train a good MoE still lives mostly in the heads of a handful of teams.
So we spent 800k GPU hours running the experiments ourselves and wrote everything down. Super happy to finally release "The Ultimate MoE Training Handbook", free and open, organized around the questions training teams hit:
- how many experts, how big, shared or not?
- why is half my step spent in all-to-all, and how do I hide it?
- why is my run 25x slower with no error in the logs?
- softmax, sigmoid or sqrt(softplus) router, and why keep it in fp32?
- aux loss or DeepSeek's loss-free bias?
- upcycle a dense checkpoint or train from scratch?
- can I serve with fewer experts per token than I trained with?
⚠️ Warning: it's long (~70k words, budget a few days, sorry) but with 55 interactive figures, code you can run and numbers from our own runs.
Read it here: huggingface.co/spaces/hugging…View quoted post on X
Love seeing dense models larger than 27b. Hope to see more development in this area.View quoted post on X
