What happens when an LLM never sees material beyond fifth grade?
We trained a 5B LittleLearner model from scratch on LittleCurriculum, a corpus restricted to K–5 material, to make this question testable.🧵
💬 Chat with LittleLearner yourself: littlelearner-ll.github.ioView quoted post on X
Transformers use the same hidden state for two competing roles: maintaining state and predicting the next token. 🔀
Splitting these roles into two separate streams outperforms standard Transformers.
Introducing the State-Prediction Separation (SPS) Transformer.
🧵View quoted post on X
[New @icmlconf paper]
Benchmarks report only a handful of aggregate metrics (e.g., overall accuracy). This aggregation hides the specifics of where a model is quietly weak, and what the benchmark itself never tests. Current evals are hence opaque and hardly actionable. (1/9)View quoted post on X
Announcing Talkie: a new, open-weight historical LLM! We trained and finetuned a 13B model on a newly-curated dataset of only pre-1930 data. Try it below!
with @AlecRad and @status_effects 🧵View quoted post on X
arxiv.org Emergent temporal abstractions in autoregressive models enable... Large-scale autoregressive models pretrained on next-token prediction and finetuned with reinforcement learning (RL) have achieved unprecedented success on many problem domains. During RL, these... Internal RL goes 🇧🇷🇧🇷🇧🇷. Catch me at the Recursive Self Improvement workshop @iclr_conf!!! @ninoscherrer and me will present our work and stick around the entire day.View quoted post on X
Poster sessions: 10:00 - 10:30 & 12:00 - 12:30 @ Room 101 - D
Let’s close the loops🚀.
