Twitxy Saved

Mustafa Shukor

@MustafaShukor1

founding rs @UMA_Robots. prev: cs phd @Sorbonne_Univ_, fair @AIatMeta, mlr @Apple, lerobot @huggingface
France mustafashukor.github.io Joined September 2021
140 Following 1.1K Followers 219 Posts

Latest posts 5

Pinned post

Mustafa Shukor @MustafaShukor1
We release a large scale study to answer the following:
- Is late fusion inherently better than early fusion for multimodal models?
- How do native multimodal models scale compared to LLMs.
- How sparsity (MoEs) can play a detrimental role in handling heterogeneous modalities? 🧵
10 replies 83 reposts 460 likes 87K views View on X
Mustafa Shukor @MustafaShukor1
Come to join the team at @UMA_Robots to build the next generation of latent-space world models for robotics !
Remi Cadene Verified account @RemiCadene Aug 18
At @UMA_Robots , we are convinced that latent-space world models will unlock scaling for robotics

1 position left for the roles

🧪 Research Scientist:
app.dover.com/apply/UMA/e223…
🚀 Research Engineer:
app.dover.com/apply/UMA/1bb6…
View quoted post on X
1 replies 3 reposts 32 likes 2.7K views View on X
Mustafa Shukor @MustafaShukor1
We have an cracked team with a strong (and long) track record of developing and shipping world models and unsupervised learning ideas for robotics.

If you want to work with us and be part of this journey, please reach out!
Pierre Sermanet Verified account @psermanet Jul 17
The SPRIND grant will accelerate UMA’s effort to develop latent world models for humanoid robots.

Throughout my career I’ve kept the objective of scaling up learning using minimal supervision and efficient data collection. This led me to propose self-supervision methods in latent space such as Time-Contrastive Networks, which demonstrated 3rd person imitation of a human entirely without labels: sermanet.github.io/imitate/

One key component was co-training different modalities, viewpoints and embodiments into a single latent space, entirely from raw unlabeled videos. The dimensions of the world discovered by the models ended up naturally aligning in latent space. This latent alignment across viewpoints and embodiments is what unlocked imitating humans directly from 3rd-person raw pixels, similarly to the concept of “mirror neurons” firing when observing someone else performing a task.

We later developed self-supervised methods with @coreylynch for generating actions by pulling together in latent space the representations of (start, goal) image pairs and sequence of images [start, …, goal]. We combined this label-free learning method with an efficient data collection method: playing.

Play is an efficient way to discover the world by leveraging existing knowledge and skills. Humans and animals use it to learn about the world and practice in advance. We called this Learning Latent Plans from Play: learning-from-play.github.io

We subsequently augmented this approach with language representations so that you could control the latent space via language conditioning. But most of the learning was coming from self-supervising on video, language labels accounted for less than 1% of data: sermanet.github.io/language-play/

I still believe that learning the world in latent space by mostly self-supervising on raw unlabeled data is the way to go. We now have stronger unsupervised learning methods, more capable hardware and better data acquisition means, it is an exciting time to keep pushing that vision.

If that vision speaks to you, we are hiring for our Embodied World Models team:

Research Scientist: app.dover.com/apply/3a4a12c9…

Research Engineer: app.dover.com/apply/3a4a12c9…
View quoted post on X
Mustafa Shukor @MustafaShukor1
We’re looking for exceptional candidates to help push the frontier of world models for robotics.

If you believe in this direction and have a strong track record, please apply here: app.dover.com/jobs/uma

Feel free to reach out if you are interested and want to know more.
Remi Cadene Verified account @RemiCadene Jul 17
Join our team to work on Embodied World-Models with @MustafaShukor1 (co-author VL-JEPA with @ylecun )

Link to apply in comments 😇
View quoted post on X