David Fan

AMI Labs

@DavidJFan

AMI Labs | ex-Meta FAIR | @Princeton CS '19
Building the next revolution of AI models that understand the real world.

New York City scholar.google.com/citations?user… Joined June 2013

517 Following 1.8K Followers 197 Posts

Latest posts 5

Pinned post

David Fan @DavidJFan Mar 4

[1/9] What happens when you treat vision as a first-class citizen during multimodal pretraining? To find out, we studied the design space of training Transfusion-style models that input and output all modalities, from scratch. Here is what we learned about visual representations, data, world modeling, architecture, and scaling behavior!

Paper: arxiv.org/abs/2603.03276
Website: beyond-llms.github.io

@TongPetersb, @DavidJFan, @__JohnNguyen__, @ellisbrown, @GaoyueZhou, @JasonQSY, @boyangzheng, @webalorn, @han_junlin, @rob_fergus, @NailaMurray, @gh_marjan, @ml_perception, Nicolas Ballas, @_amirbar, Michael Rabbat, Jakob Verbeek, @LukeZettlemoyer, @koustuvsinha, @ylecun, @sainingxie

arxiv.org Beyond Language Modeling: An Exploration of Multimodal Pretraining The visual world offers a critical axis for advancing foundation models beyond language. Despite growing interest in this direction, the design space for native multimodal models remains opaque....

David Fan @DavidJFan Aug 7

Nice work Junlin!!

Junlin Han @han_junlin Aug 6

We live in a multimodal world. We see, talk, act, and dream.

Yet most LLMs still start with language pretraining. Why not train them natively with multimodal I/O from scratch?

Because it’s SUPER HARD, adding modalities triggers training instability, design complexity, and often modal competition

So what’s the path forward?

Introducing: Towards Physics of Multimodal Pretraining (junlinhan.github.io/projects/physi…)

We unpack the underlying mechanics of multimodal pretraining across 4 aspects: Knowledge Flow, Modality Synergy, Early Unification, and Recipe.

View quoted post on X

David Fan @DavidJFan Jul 26

Nice work!! Really glad to hear WebSSL works well for robot policies 😁 Representation matters

Jeff Cui Verified account @jeffacce Jul 22

Your policy doesn't need 7B params. It simply needs dense features.

Introducing Patch Policy: pretrained ViT + small transformer beats OpenVLA-OFT with 0.7% of its params, and trains on a 5090.

Here it inserts a cable (~2mm tol), and does it again as we unplug mid-rollout. 🧵

View quoted post on X

David Fan @DavidJFan Jul 7

@__JohnNguyen__ and I are presenting the Beyond Language Modeling paper as an ICML spotlight in 30 minutes at the 10:30 AM poster session!

I’ll also be at the AMI Mixer on Thursday and hanging around in Seoul until early next week :)

David Fan Verified account @DavidJFan Mar 4

[1/9] What happens when you treat vision as a first-class citizen during multimodal pretraining? To find out, we studied the design space of training Transfusion-style models that input and output all modalities, from scratch. Here is what we learned about visual representations, data, world modeling, architecture, and scaling behavior!

Paper: arxiv.org/abs/2603.03276
Website: beyond-llms.github.io

@TongPetersb, @DavidJFan, @__JohnNguyen__, @ellisbrown, @GaoyueZhou, @JasonQSY, @boyangzheng, @webalorn, @han_junlin, @rob_fergus, @NailaMurray, @gh_marjan, @ml_perception, Nicolas Ballas, @_amirbar, Michael Rabbat, Jakob Verbeek, @LukeZettlemoyer, @koustuvsinha, @ylecun, @sainingxie
View quoted post on X

David Fan @DavidJFan Apr 16

Congrats Dr. @TongPetersb!! Your research journey has clearly culminated in a very cohesive and inspiring narrative that unifies several areas of work with much scope for future expansion :D It's a testament to your work ethic, good taste in problems, and attention to detail. I'm so proud of you! Very lucky to learn from you

John Nguyen Verified account @__JohnNguyen__ Apr 16

Congrats Dr. Tong!
Really glad to be a part of your PhD journey @TongPetersb

View quoted post on X