Junlin Han

We live in a multimodal world. We see, talk, act, and dream.

Yet most LLMs still start with language pretraining. Why not train them natively with multimodal I/O from scratch?

Because it’s SUPER HARD, adding modalities triggers training instability, design complexity, and often modal competition

So what’s the path forward?

Introducing: Towards Physics of Multimodal Pretraining (junlinhan.github.io/projects/physi…)

We unpack the underlying mechanics of multimodal pretraining across 4 aspects: Knowledge Flow, Modality Synergy, Early Unification, and Recipe.

Nice model with native multimodality!!! Congrats to the team cooking it!

I joined @amilabs ~a month ago as an MTS, working on world model pre-training and capability emergence.

There has never been a better time to build AI for the real world with a scaling mindset. Thrilled to join forces with friends old and new to build the next frontier🚀!

Computer vision is officially 3D only.

7 out of the 10 candidates are 3D. Multimodal foundation models, generative scaling, representation learning? Irrelevant to intelligence. The universe is made of point clouds, meshes, and Gaussians.

(Nothing against the authors btw)

Thanks for posting our work! Hope this can be helpful for multimodal intelligence!