Twitxy Saved

Jiatao Gu @ CoRL2026

Verified account AMI Labs

@thoma_gu

Assistant Prof @CIS_Penn & MTS @amilabs | Prev: @Apple MLR @Meta FAIR | PhD @HKUniversity | Research on Generative AI & World Models. また、日本語もできます。
Philadelphia, PA jiataogu.me Joined October 2012
2.3K Following 8.6K Followers 790 Posts

Latest posts 5

Pinned post

Jiatao Gu @ CoRL2026 Verified account @thoma_gu
We’re teaching a new course at Penn: CIS 6280 · World Models 🌍
cis.upenn.edu/~cis6280/

Every day, LLMs amaze me with one more thing they can solve. But I’ve always felt there’s more to learn than what humans have written down—from observing the world, interacting with it, and experiencing what happens next. To me, that’s what world models are about—and why I see them as one of AI’s next big bets.

However, I struggled to find a systematic course on them—so we are building one!

Topics will include: representation learning, generative models, simulation, model-based RL, video and 3D generation, world models for robotics, reasoning, and code-based world models.

Hands-on work will include:
- Building an environment
- Training a world model
- Learning a policy
and a final research project with leaderboard.

We’re already six lectures in. Slides, demos, and readings are public, and we’ll keep adding materials throughout the fall.

Big thanks to our TA team— @TongMutianTMT @hagsaeng_bag @EnxinSong @KeelyAi04 —for helping bring this course to life!
58 replies 368 reposts 2.4K likes 578K views View on X
Jiatao Gu @ CoRL2026 Verified account @thoma_gu
Excited to share OmniGuide led by @TimSong52005757 at #CoRL2026 — test-time guidance for diffusion-based VLAs. We can turn any VLM, geometry, and human guidance into a differentiable 3D energy function that steers action sampling of a frozen VLA. Poster Mon 11/9, 5–6:30pm, #32.
TimSong✈️CoRL Verified account @TimSong52005757 6d
VLAs like π0.5 and GR00T still struggle: collisions in clutter, complex semantics, learning from human demos.
OmniGuide steers any flow/diffusion VLA at test time with VLM agents, geometry foundation models & human teaching.
No extra data, no retraining, composable.
#CoRL2026
View quoted post on X
3 replies 0 reposts 30 likes 1.8K views View on X
Jiatao Gu @ CoRL2026 Verified account @thoma_gu
Super cool! open-weight video model with faster-than-realtime generation!! Cannot wait to try!
Pika @pika_labs Oct 29
Meet Pika H3 Flex, debuting today as the #1 open weight model on DesignArena’s t2v board.

It’s built for professional iteration, with exportable audio stems, resolutions up to 4k, and faster-than-realtime generation. It can produce a 15-second clip with soundtrack in as little as 7 seconds.

So, yeah. It’s kind of a flex 💅
View quoted post on X
5 replies 0 reposts 12 likes 2.8K views View on X
Jiatao Gu @ CoRL2026 Verified account @thoma_gu
Announcing ProgramLab: 1,261 real-world programs for agents to rebuild through trial-and-error RL, from CLI tools to 100K+ line projects. We can train single and multi agents on it, which improves ProgramBench pass rates.
Many thanks to my PhD student @tyao923 for leading this!
Yao Tang Verified account @tyao923 Oct 28
When an agent meets a program: 𝗣𝗹𝗮𝘆 𝗶𝘁. 𝗥𝗲𝗯𝘂𝗶𝗹𝗱 𝗶𝘁.

𝗣𝗿𝗼𝗴𝗿𝗮𝗺𝗟𝗮𝗯 gives agents 1,261 real-world programs to practice on.

𝗧𝗿𝗮𝗶𝗻 𝗼𝗻𝗲 𝗮𝗴𝗲𝗻𝘁, 𝗼𝗿 𝗮 𝘄𝗵𝗼𝗹𝗲 𝘁𝗲𝗮𝗺.

🧵0/n
View quoted post on X
4 replies 4 reposts 58 likes 3.9K views View on X
Jiatao Gu @ CoRL2026 Verified account @thoma_gu
Excited to see more cool work built on TARFlow (arxiv.org/abs/2412.06329) for lossless invertible representations that improves training.

Related: our recent STARFlow2 also explored combining a VLM with TARFlow for unified multimodal generation arxiv.org/abs/2605.08029
arxiv.org Normalizing Flows are Capable Generative Models Normalizing Flows (NFs) are likelihood-based models for continuous inputs. They have demonstrated promising results on both density estimation and generative modeling tasks, but have received...
Suchir Agarwal Verified account @agarwal_suchir Oct 19
1/8 Introducing LILIE: a Language-Informed and Lossless Image Encoder.

We train a language-informed encoder to supervise end-to-end visual generative models. Against a pixel-space JiT model, LILIE improves gFD on GPIC by 14 points (66.84 -> 53.01).
View quoted post on X