Stephen James

𝗔𝗳𝘁𝗲𝗿 𝟭𝟬+ 𝘆𝗲𝗮𝗿𝘀 𝗶𝗻 𝗿𝗼𝗯𝗼𝘁 𝗹𝗲𝗮𝗿𝗻𝗶𝗻𝗴, from my PhD at Imperial to Berkeley to building the Dyson Robot Learning Lab, one frustration kept hitting me:

𝗪𝗵𝘆 𝗱𝗼 𝗜 𝗵𝗮𝘃𝗲 𝘁𝗼 𝗿𝗲𝗯𝘂𝗶𝗹𝗱 𝘁𝗵𝗲 𝘀𝗮𝗺𝗲 𝗶𝗻𝗳𝗿𝗮𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲 𝗼𝘃𝗲𝗿 𝗮𝗻𝗱 𝗼𝘃𝗲𝗿 𝗮𝗴𝗮𝗶𝗻?

𝗧𝗵𝗲 𝗽𝗮𝘁𝘁𝗲𝗿𝗻 𝗜 𝗸𝗲𝗽𝘁 𝘀𝗲𝗲𝗶𝗻𝗴:

• New robotics team starts
• Spends 6 months building data collection pipeline
• Spends another 3 months debugging synchronization issues
• Finally starts collecting task-specific data
• Realizes their infrastructure choices limit their flexibility
• Starts over

𝗧𝗵𝗶𝘀 𝗶𝘀 𝘁𝗵𝗲 𝘄𝗵𝗼𝗹𝗲 𝗽𝗼𝗶𝗻𝘁 𝗼𝗳 𝗿𝗼𝗯𝗼𝘁 𝗹𝗲𝗮𝗿𝗻𝗶𝗻𝗴: Robot learning is fundamentally data-driven. Whether you're picking strawberries or assembling electronics, the core infrastructure needs are identical. That's actually why I was so interested in pursuing data-driven robotics over a decade ago.

𝗬𝗼𝘂 𝗮𝗹𝘄𝗮𝘆𝘀 𝗻𝗲𝗲𝗱:

• Multi-sensor data synchronization across different frequencies
• Flexible storage that works with future algorithms
• Visualization tools to understand your data
• The ability to experiment with different temporal resolutions
• Robust logging that captures everything you might need later

The trend towards AI in robotics is growing, with robots needing to process and analyze large amounts of sensor data to manage variability and unpredictability in real environments.

𝗕𝘂𝘁 𝗲𝘃𝗲𝗿𝘆 𝘁𝗲𝗮𝗺 𝗯𝘂𝗶𝗹𝗱𝘀 𝘁𝗵𝗶𝘀 𝗳𝗿𝗼𝗺 𝘀𝗰𝗿𝗮𝘁𝗰𝗵. Imagine if every web developer had to build their own database, web server, and deployment pipeline before writing their first line of application code.

𝗧𝗵𝗶𝘀 𝗶𝘀 𝘄𝗵𝘆 𝗜 𝗳𝗼𝘂𝗻𝗱𝗲𝗱 𝗡𝗲𝘂𝗿𝗮𝗰𝗼𝗿𝗲.

Instead of every robotics team spending months on infrastructure, we provide the common tools that let you go from "I have a robot" to "I'm shipping intelligent robot behaviors" in days, not months.

𝗧𝗵𝗲 𝗿𝗲𝗮𝗹 𝗶𝗻𝗻𝗼𝘃𝗮𝘁𝗶𝗼𝗻 𝗶𝗻 𝗿𝗼𝗯𝗼𝘁𝗶𝗰𝘀 𝘄𝗼𝗻'𝘁 𝗰𝗼𝗺𝗲 𝗳𝗿𝗼𝗺 𝗲𝘃𝗲𝗿𝘆𝗼𝗻𝗲 𝗿𝗲𝗯𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝘁𝗵𝗲 𝘀𝗮𝗺𝗲 𝗽𝗹𝘂𝗺𝗯𝗶𝗻𝗴. 𝗜𝘁'𝗹𝗹 𝗰𝗼𝗺𝗲 𝗳𝗿𝗼𝗺 𝘁𝗲𝗮𝗺𝘀 𝘄𝗵𝗼 𝗰𝗮𝗻 𝗳𝗼𝗰𝘂𝘀 𝗲𝗻𝘁𝗶𝗿𝗲𝗹𝘆 𝗼𝗻 𝘄𝗵𝗮𝘁 𝗺𝗮𝗸𝗲𝘀 𝘁𝗵𝗲𝗶𝗿 𝗮𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻 𝘂𝗻𝗶𝗾𝘂𝗲.

Robot learning shouldn't be bottlenecked by infrastructure. It should be bottlenecked by creativity.

What's the longest you've spent building infrastructure before getting to the actual robotics problem you wanted to solve?

A popular VLA training recipe blocks the action expert from updating the VLM underneath it. A new paper asks whether that should be the default.

Asher Hancock, Marco Pavone and the team at @nvidia, Princeton University and Stanford University varied two things independently: whether the VLM also predicts discrete action tokens, and whether the expert’s gradients flow back into it.

With action tokens on, blocking the gradients costs little in open-loop driving. With them off, letting the gradients through cuts trajectory error by about 25%.

Closed loop is where it gets interesting. Trained together from the start, letting the gradients through lifted driving route progress from 0.600 to 0.703, even with action tokens on. In manipulation, a π0.5 re-implementation went from 10.3% to 32.9% task progress without FAST, and from 19.5% to 30.7% with it. MolmoAct2 moved the same direction, though the gain with FAST was small (22.0% to 24.3%).

One training run per condition, and all the closed-loop results are in simulation. I’d read it as a good reason to question insulation as the default, not proof it’s wrong.

Paper: arxiv.org/abs/2611.05249
Project page: mot-vla-gradient-routing.github.io

Wrist cameras give robots great close-ups, but they’re stuck to the hand. FoveaVLA asks whether the policy can just choose where to look instead.

@bingao_chen, Karen Liu, Guy Tevet and the team at @Stanford and @Princeton built it on π0.5. With every action chunk, the model also predicts the box it will need to see next and crops it from the full-resolution external image. Squashing a 1280x720 frame to 224x224 blurs the gripper and the object, and the crop gives that detail back.

One external camera plus depth, no wrist camera: 34 of 48 real-world trials (70.8%), against 25 of 48 (52.1%) for wrist-equipped π0.5. Retrain π0.5 on the external view alone and it falls to 6.3%.

It’s a small eval, 12 trials per suite, so I’d like to see more. The ablation is the part that convinces me, though. Crop alone gets 20.8%, depth alone 6.3%, both together 70.8%.

Code and models are open and can be found below:

Paper: lnkd.in/egBJndZF
Project page: lnkd.in/evwNNUbf

The @Neuracore_AI team will be running live demos at @ABBRobotics upcoming event on the 2nd of December.

Come and join us and see what’s ready to deploy on the robots you already run.

Good data starts with good visibility.

Trace Viewer makes it easier to understand what's happening during data collection, spot issues early and know exactly what you're feeding into training.

Another step towards making robot learning workflows more reliable.