My PhD thesis is now available on DCU's DORAS!
Give it a read and let me know what you think!
doras.dcu.ie/33491/
Give it a read and let me know what you think!
doras.dcu.ie/33491/
Mohammed Sabry
Verified account This account is verified. Learn more@M___Sabry
600 Following 219 Followers 745 Posts
Latest posts 5
"deep learning is the geometric mean of biology and physics"
- Ilya Sutskever (2020)
- Ilya Sutskever (2020)
This needs a mechanistic look...the shorter recovery period points to a circuit accessibility problem....also neural nets develop many redundant circuits, which points more toward the circuits being "harder to activate" than being damaged
1/
During OLMo3-32B pre-training, accuracy on an arithmetic diagnostic hits 81% at 2.17T tokens, collapses to 0% at 2.19T, and jumps back to 81.7% at 2.21T.
All while validation loss decreases smoothly.
The phenomenon: mode-hopping. 🧵View quoted post on X
⚡⚡⚡
We still see a big room for improvement on top of the current Mellum2 base, but we wanted to share the results with you as soon as possible. We're planning to scale further both post-training AND pre-training, so look forward to what the team ships next 😉View quoted post on X
A good question from here is whether there is an optimal context length, and whether it follows a predictable scaling law (figure 2 and 9 do not hint at that in any way, which is reasonable given the contribution's angle)
arxiv.org/abs/2608.12218
During pretraining if long context gives enough information for tasks then the model could learn to utilize context instead of internalizing tasks. Maybe internalizing tasks could be suppressed and it could be useful to build a pretraining curriculum?View quoted post on X
