Twitxy Saved

Hu Xu (back on robotics)

Verified account This account is verified. Learn more

@Hu_Hsu

FAIR, @AIatMeta, #MetaCLIP (scaling #CLIP data from scratch as the #imagenet moment for web) #llama 3, DINO, SAM3, PE; coined #posttraining in 2019 #AI #GenAI
Bellevue, WA ai.meta.com/people/7761982… Joined October 2014
737 Following 1.2K Followers 302 Posts

Latest posts 5

Pinned post

Hu Xu (back on robotics) Verified account @Hu_Hsu
The cross-modal multilingual capability in Meta CLIP 2 is naturally from scaling. It is a reflection of the Bitter Lesson(@RichardSSutton) on building a scalable learning environment, and let optimization and scale do the heavy lifting. Inspired by Jensen Huang’s scaling law plot (at CES2025), the scaling isn’t just about increasing compute or data at each stage—it’s about consistently identifying bottlenecks, removing constraints, and rethinking a simpler setup to make it more scalable. This mindset extends to data curation itself: starting from ImageNet to worldwide data curation. @AIatMeta
Jason Weston Verified account @jaseweston Jul 30, 2025
🌿Introducing MetaCLIP 2 🌿
📝: arxiv.org/abs/2507.22062
code, model: github.com/facebookresear…

After four years of advancements in English-centric CLIP development, MetaCLIP 2 is now taking the next step: scaling CLIP to worldwide data. The effort addresses long-standing challenges: (1) large-scale non-English data curation pipelines are largely undeveloped, and (2) the curse of multilinguality, where English performance often degrades in multilingual CLIP compared to English-only CLIP. With a complete recipe for worldwide CLIP—spanning data curation, modeling, and training—we show that English and non-English worlds can mutually benefit and elevate each other, achieving SoTA multilingual performance. Join the Meta booth at #ACL2025 to learn more.
(1/3)
View quoted post on X
3 replies 6 reposts 23 likes 15K views View on X
Hu Xu (back on robotics) Verified account @Hu_Hsu
@gabriberton Thx for explaining MLLM ~= encoder and NMM ~= encoder-free. I agree with all the advantages of your MLLM case. And NMM has its advantages, too. We unlikely have a binary answer to this A vs B question (the answer depends on the use cases). That's why we study encoding (some principles need to hold) instead of 'encoder'. 😋
Gabriele Berton Verified account @gabriberton May 8
Cool paper from Meta suggesting that future MLLMs will be Native Multimodal Models (NMM), hence no vision encoders anymore

But I disagree

I actually think we'll go in the other direction (what? more encoders? yes! read on...)

All you need to know about the future of MLLMs 🧵
View quoted post on X
0 replies 0 reposts 13 likes 698 views View on X
Hu Xu (back on robotics) Verified account @Hu_Hsu
LLMs have made remarkable progress, yet MLLMs still face foundational challenges on conflicting training recipes and scaling: how should visual information flow through a LLM?

We show that neither the standalone ViTs that start from the vision side, nor encoder-free designs start from the language side is all right. In our controlled study, we revisit the architectural choices that reshape the choice of whether encoder or not to: 'encoding' and where to train encoding. We hope this work opens up new directions for both vision and multimodal research.

Super excited to work on this with
@_ellisbrown @choang333 @JasonQSY @ShangwenLi1 @ml_perception @gh_marjan @LukeZettlemoyer @giffmana @mengyer Jakob Verbeek @sainingxie @AIatMeta
Chris Hoang Verified account @choang333 3d
Labs disagree on vision encoders for MLLMs! ⚔️

Gemma 4 12B scrapped the encoder.
Kimi K3 dropped encoder pretraining.
Inkling 2 Small revived the encoder Inkling killed.

We trained 200+ models in a controlled study to settle it:
✅ Keep vision parameters, skip the pretraining. And they don't have to be an encoder! 🤔

Paper: arxiv.org/abs/2611.17334 🧵
View quoted post on X
2 replies 1 reposts 21 likes 1.1K views View on X
Hu Xu (back on robotics) Verified account @Hu_Hsu
This work builds on years of research at @metaai, from curating MetaCLIP data to rethinking vision encoding.

Although the data is more invariant and offers observations about the physical world, how to train an architecture that carries those observations into an LLM is still an open question. In a strictly controlled study with total compute, we explored both standalone ViT encoders and encoder-free designs. Such binary decision on vision alone is not enough. We rethink how vision encoding should place its capacity and flow naturally with the LLM.

Super excited to work on this with @_ellisbrown @choang333, @JasonQSY
@ShangwenLi1
@ml_perception
@gh_marjan
@LukeZettlemoyer
@giffmana
@mengyer
Jakob Verbeek
@sainingxie
Ellis Brown Verified account @_ellisbrown 3d
how should multimodal models encode visual input?

→ pretrained encoder?
→ encoder-free?
→ before the LLM?

published models bundle these choices together. we trained 200+ models from scratch to see which of these choices matters

only one of them does

🧵 [1/n]
View quoted post on X
13 replies 3 reposts 68 likes 2.1K views View on X