Pinned post
MIT NLP
Verified account This account is verified. Learn more@nlp_mit
NLP Group at @MIT_CSAIL! PIs: @yoonrkim @jacobandreas @lateinteraction @pliang279 @david_sontag, Jim Glass, @roger_p_levy
74 Following 5K Followers 151 Posts
Latest posts 5
check out @akarshkumar0101’s awesome interview on @MLStreetTalk!!!
"Evolution is anything but random."
This is INSANE...
@akarshkumar0101 (@MIT_CSAIL, @SakanaAILabs) on artificial life: searching the space of possible universes with foundation models, why evolved networks find structure SGD misses (with @kenneth0stanley), and LLMs evolving Core War programs.View quoted post on X
@SophieLWang and co explain what might actually be driving reasoning gains!
Can “chicken” make a base model reason? 🐔
Yes!
Why? With the right first tokens, base models can match RL in reasoning. We trace this to learned training data associations and use a simple data edit to make “chicken” elicit reasoning too.
blog: sophielwang.com/cuesView quoted post on X
let’s get linear attention to be more expressive!
Linear attention is arguably the most naive RNN, but still massively outperforms traditional RNNs by maintaining a matrix state. So... what if we use a (triadic) outer product of three vectors and maintain a three-dimensional state?
Introducing: Triadic Linear Attention 🧵View quoted post on X
We do teacher forcing on input tokens, why not do it for RNN memory states as well?
Pretraining Recurrent Networks without Recurrence by @akarshkumar0101 & @phillip_isola is a great paper!
It makes an end-run around the problems of RNNs via a transformer teacher to learn good predictive state representations & supervised learning of a memory transition functionView quoted post on X
