Twitxy Saved

Pierre Sermanet

Verified account This account is verified. Learn more

@psermanet

Co-Founder & Chief Scientist at UMA
sermanet.github.io Joined November 2017
274 Following 6.4K Followers 117 Posts

Latest posts 5

Pierre Sermanet Verified account @psermanet
Great to hang out with peers at @corl_conf, next CoRL in Austin the robots will be the ones bull riding?
Louis Le Lay Verified account @LouisLeLay4 6d
CoRL was a blast, met a ton of great people!

To celebrate, I taught a G1 to ride a mechanical bull the Texan way 🤠🐂 Inspired by @the_real_btaba's demo at the MuJoCo meetup, but this time in mjlab and holding a rope! It's already better at it than I am...
View quoted post on X
Pierre Sermanet Verified account @psermanet
The SPRIND grant will accelerate UMA’s effort to develop latent world models for humanoid robots.

Throughout my career I’ve kept the objective of scaling up learning using minimal supervision and efficient data collection. This led me to propose self-supervision methods in latent space such as Time-Contrastive Networks, which demonstrated 3rd person imitation of a human entirely without labels: sermanet.github.io/imitate/

One key component was co-training different modalities, viewpoints and embodiments into a single latent space, entirely from raw unlabeled videos. The dimensions of the world discovered by the models ended up naturally aligning in latent space. This latent alignment across viewpoints and embodiments is what unlocked imitating humans directly from 3rd-person raw pixels, similarly to the concept of “mirror neurons” firing when observing someone else performing a task.

We later developed self-supervised methods with @coreylynch for generating actions by pulling together in latent space the representations of (start, goal) image pairs and sequence of images [start, …, goal]. We combined this label-free learning method with an efficient data collection method: playing.

Play is an efficient way to discover the world by leveraging existing knowledge and skills. Humans and animals use it to learn about the world and practice in advance. We called this Learning Latent Plans from Play: learning-from-play.github.io

We subsequently augmented this approach with language representations so that you could control the latent space via language conditioning. But most of the learning was coming from self-supervising on video, language labels accounted for less than 1% of data: sermanet.github.io/language-play/

I still believe that learning the world in latent space by mostly self-supervising on raw unlabeled data is the way to go. We now have stronger unsupervised learning methods, more capable hardware and better data acquisition means, it is an exciting time to keep pushing that vision.

If that vision speaks to you, we are hiring for our Embodied World Models team:

Research Scientist: app.dover.com/apply/3a4a12c9…

Research Engineer: app.dover.com/apply/3a4a12c9…
Remi Cadene Verified account @RemiCadene Jul 17
Join our team to work on Embodied World-Models with @MustafaShukor1 (co-author VL-JEPA with @ylecun )

Link to apply in comments 😇
View quoted post on X
3 replies 7 reposts 37 likes 10K views View on X
Pierre Sermanet Verified account @psermanet
At UMA, we build and own the full stack, from hardware to software.
This allows us to bake safety in at all levels, rather than bolt it on as an afterthought.
Because trust is everything when robots share our world.
Reach out if you want to be part of this journey.
3 replies 21 reposts 69 likes 19K views View on X