Soham De
Verified account This account is verified. Learn more@sohamde_
Training at OpenAI. Prev: Research Scientist at DeepMind, PhD at the University of Maryland.
1.2K Following 3.2K Followers 193 Posts
Latest posts 5
Our new paper sheds light on the process of knowledge acquisition in language models, with implications for
- data curricula
- the challenges of learning new knowledge when fine-tuning
- the emergence of hallucinations.
Nicolas did a great job on the project! See his thread👇
- data curricula
- the challenges of learning new knowledge when fine-tuning
- the emergence of hallucinations.
Nicolas did a great job on the project! See his thread👇
Large language models store vast amounts of knowledge, but how exactly do they learn it?View quoted post on X
Excited to share my @GoogleDeepMind internship results, which reveal the fascinating dynamics behind factual knowledge acquisition in LLMs!
arxiv.org/abs/2503.21676
Two months back, we released a 9B RecurrentGemma model, one of the strongest SSM-based language models out there, trained on 2T tokens!
I finally updated arXiv with some of our results: arxiv.org/abs/2404.07839
Link to weights and code for our models in thread!
I finally updated arXiv with some of our results: arxiv.org/abs/2404.07839
Link to weights and code for our models in thread!
It was fun to moderate this discussion with a great group of panelists. Lots of interesting points made on how to approach the next gen of seq modelling architectures. Thanks for the invite @caglarml @orvieto_antonio Razvan and others!
The panel discussion at NGSM workshop going on full steam ahead with a great line of panelists moderated by @sohamde_ ...View quoted post on X
🔥 Introducing our 9B language model, trained on 2 trillion tokens! 🚀
Based on Griffin (arxiv.org/abs/2402.19427) and delivers:
💪 Powerful performance
⚡️ Lightning-fast inference
Pretrained and instruction-tuned models now available on HF & Kaggle!
Start building today! 🏗️
Based on Griffin (arxiv.org/abs/2402.19427) and delivers:
💪 Powerful performance
⚡️ Lightning-fast inference
Pretrained and instruction-tuned models now available on HF & Kaggle!
Start building today! 🏗️
RecurrentGemma-9B is out!View quoted post on X
kaggle.com/models/google/…
huggingface.co/google/recurre…
- Uses Griffin architecture, combining linear recurrence with local attention
- Downstream evals comparable to Mistral and Gemma
- Faster inference, especially for long sequences or large batch sizes
1/n