Pinned post
I had an incredible time training this model. tldr on some nonobvious things I learned:
- Dense models can be surprisingly competitive with much large MoEs in downstream evals! Matching perplexity doesn't necessarily mean matching downstream capability; our dense model often performs better, due to a variety of factors (attention density, less memorization, etc.)
- Once you solve your infrastructure and get your synthetic environment pipelines working well, mixed RL is actually extremely efficient at improving your model. Can't wait to continue scaling posttraining!
- You can use swarms of small agentic models to turn qualitative questions about your datasets into programmable measurements. This creates a powerful feedback loop for systematically improving data quality and climbing tasks
- Dense models can be surprisingly competitive with much large MoEs in downstream evals! Matching perplexity doesn't necessarily mean matching downstream capability; our dense model often performs better, due to a variety of factors (attention density, less memorization, etc.)
- Once you solve your infrastructure and get your synthetic environment pipelines working well, mixed RL is actually extremely efficient at improving your model. Can't wait to continue scaling posttraining!
- You can use swarms of small agentic models to turn qualitative questions about your datasets into programmable measurements. This creates a powerful feedback loop for systematically improving data quality and climbing tasks
Today we release Ripple-65B Preview, our 65B dense LLM we pretrained from scratch.
Ripple sets a new state-of-the-art for U.S. open-weight models in agentic coding tasks, surpassing significantly larger models including Muse Spark Open-816B, Beam-501B, Inkling-975B, and Nemotron 3 Ultra-550B.
Learn more: ricursive.com/rippleView quoted post on X
