Aurora Feng
Verified account This account is verified. Learn more@aurorafeng_01
founder & ceo @neuralmotion
cofounder @saturdayrobotic
robotics, world model, in-context learning
@stanford ‘24 🌲
cofounder @saturdayrobotic
robotics, world model, in-context learning
@stanford ‘24 🌲
310 Following 951 Followers 148 Posts
Latest posts 5
honored to be a part of the conference! can’t wait for what’s next
The first Humanoid Hub Conference was incredible.View quoted post on X
Pulling off an event like this is far harder than I imagined, and it would not have been possible without the people around me.
Thanks to @itsajchan and the team, who carried a huge amount of this.
It was fun hosting @saturdayrobotic Reading Club 34 with @AIMSafety_ !
A big thank you to @Skye_Kim_25 for sponsoring this week's session and kicking off with a discussion around safety for phsyical AI, a super important but underrated topic, and the benchmarks they've been developing such as EgoSafetyBench. One particularly interesting finding: across 50 completed task attempts in their evaluation, only 7 were completed safely. Task success and safe execution are two very different metrics, especially as we move toward deploying increasingly autonomous robots in the real world.
Next, @wangyenjen from @UCBerkeley share his latest work: Reconstruct, Practice, Go Real (RPG): How Embodied Agents Improve Without Retraining Weights. What if we could improve a robot’s capabilities by improving the tools and skills available to its agent, rather than the model itself?
1. What is the right level of abstraction for a robot’s skill library? Should skills be low-level control primitives or higher-level behaviors?
RPG uses a hierarchical skill library, ranging from basic operations such as controlling the gripper and end-effector pose to higher-level behaviors such as grasping an object or identifying keypoints for cloth folding. If computation and API costs were unlimited, an agent could theoretically solve many tasks by repeatedly composing low-level primitives. The tradeoff is efficiency: a more abstract skill library allows the agent to reuse previously solved behaviors instead of reasoning through every action from scratch.
2. Why use coding agents for self-improvement instead of training a conventional robot policy?
One motivation is that coding agents can modify their own executable tools. When a policy fails, conventional robot learning often requires additional training data or gradient updates. RPG instead asks whether the agent can address the failure by changing its available skills, improving its perception or planning logic, or introducing a new reusable behavior. The resulting improvement becomes available to subsequent tasks through the shared skill library, and this shifts the unit of improvement from model parameters to executable capabilities.
3. Does the improved skill library generalize to unseen tasks, different robots, and different foundation models?
Interestingly, changing the runtime language model after self-improvement did not substantially affect task success rates among the capable models tested. After 15 rounds of practice, the system had accumulated a sufficiently comprehensive set of instructions and executable skills that the runtime model no longer needed to solve every problem independently. This suggests a potentially interesting division of labor: more capable models can be used during the expensive skill-discovery process, while deployment can rely on a reusable library and a less demanding agent.
A big thank you to @Skye_Kim_25 for sponsoring this week's session and kicking off with a discussion around safety for phsyical AI, a super important but underrated topic, and the benchmarks they've been developing such as EgoSafetyBench. One particularly interesting finding: across 50 completed task attempts in their evaluation, only 7 were completed safely. Task success and safe execution are two very different metrics, especially as we move toward deploying increasingly autonomous robots in the real world.
Next, @wangyenjen from @UCBerkeley share his latest work: Reconstruct, Practice, Go Real (RPG): How Embodied Agents Improve Without Retraining Weights. What if we could improve a robot’s capabilities by improving the tools and skills available to its agent, rather than the model itself?
1. What is the right level of abstraction for a robot’s skill library? Should skills be low-level control primitives or higher-level behaviors?
RPG uses a hierarchical skill library, ranging from basic operations such as controlling the gripper and end-effector pose to higher-level behaviors such as grasping an object or identifying keypoints for cloth folding. If computation and API costs were unlimited, an agent could theoretically solve many tasks by repeatedly composing low-level primitives. The tradeoff is efficiency: a more abstract skill library allows the agent to reuse previously solved behaviors instead of reasoning through every action from scratch.
2. Why use coding agents for self-improvement instead of training a conventional robot policy?
One motivation is that coding agents can modify their own executable tools. When a policy fails, conventional robot learning often requires additional training data or gradient updates. RPG instead asks whether the agent can address the failure by changing its available skills, improving its perception or planning logic, or introducing a new reusable behavior. The resulting improvement becomes available to subsequent tasks through the shared skill library, and this shifts the unit of improvement from model parameters to executable capabilities.
3. Does the improved skill library generalize to unseen tasks, different robots, and different foundation models?
Interestingly, changing the runtime language model after self-improvement did not substantially affect task success rates among the capable models tested. After 15 rounds of practice, the system had accumulated a sufficiently comprehensive set of instructions and executable skills that the runtime model no longer needed to solve every problem independently. This suggests a potentially interesting division of labor: more capable models can be used during the expensive skill-discovery process, while deployment can rely on a reusable library and a less demanding agent.
I’ll be speaking at the SF Humanoid Hub Conference tomorrow 10/15! Thanks @TheHumanoidHub for inviting me😆
Excited to host the upcoming @saturdayrobotic reading club 34 on 10/17 with @AIMSafety_ ! luma.com/oq82q5ax
Guest speaker: @wangyenjen from @UCBerkeley will talk about his newest work: RPG: How Robots Improve Without Retraining, a framework for autonomous improvement of robot execution systems without updating model weights.
Sponsor highlight: Al agents are making decisions, calling APls, and taking actions, with no one watching. AIM Intelligence attacks your Al before real attackers do, and enforces real-time guardrails to keep every agent under your control.
See you guys this Sat! @junfanzhu98 @solsol_choi Sky Kim @jerryhuang01 @RoboticsCtr
Guest speaker: @wangyenjen from @UCBerkeley will talk about his newest work: RPG: How Robots Improve Without Retraining, a framework for autonomous improvement of robot execution systems without updating model weights.
Sponsor highlight: Al agents are making decisions, calling APls, and taking actions, with no one watching. AIM Intelligence attacks your Al before real attackers do, and enforces real-time guardrails to keep every agent under your control.
See you guys this Sat! @junfanzhu98 @solsol_choi Sky Kim @jerryhuang01 @RoboticsCtr