a simulation can look photorealistic and still be useless for training a robot.
this confused me for a long time, so here is the clean version.
when you look at a game, you judge the graphics. light, shadows, reflections, texture detail. that is rendering - the job of producing a convincing image for a human eye.
a robot does not care about the image being beautiful. it cares about whether the physics lies.
three things decide that, and none of them are visual.
● friction - does the object slide out of the gripper at the same angle it would in reality, or does the sim hold it like glue?
● inertia - when the arm swings a weight, does the weight pull back with the right force, or does it behave like it is made of foam?
● deformation - does a towel crumple, a cable bend and a sponge squash the way real ones do, or do they move like rigid plastic?
a model trained in a sim where friction is wrong learns a grip that drops things in the real world. the picture was perfect. the lesson was false.
this is the sim-to-real gap, and it is not a graphics problem. you can close it with a flat grey render that gets the forces right, and you can completely fail with a cinematic render that gets them wrong.
the practical consequence is uncomfortable. you cannot verify a simulation by looking at it. a human looking at a render has no way to tell whether the friction coefficient is correct. the only check is to run the same task on real hardware and compare what happened.
which is exactly why real-world recorded episodes stay valuable even in a world full of simulators. the sim is where you get volume. the real data is the only thing that tells you the sim is lying.
beauty is not fidelity. i think that one sentence saves people a lot of wasted compute.
@prismaxai
this confused me for a long time, so here is the clean version.
when you look at a game, you judge the graphics. light, shadows, reflections, texture detail. that is rendering - the job of producing a convincing image for a human eye.
a robot does not care about the image being beautiful. it cares about whether the physics lies.
three things decide that, and none of them are visual.
● friction - does the object slide out of the gripper at the same angle it would in reality, or does the sim hold it like glue?
● inertia - when the arm swings a weight, does the weight pull back with the right force, or does it behave like it is made of foam?
● deformation - does a towel crumple, a cable bend and a sponge squash the way real ones do, or do they move like rigid plastic?
a model trained in a sim where friction is wrong learns a grip that drops things in the real world. the picture was perfect. the lesson was false.
this is the sim-to-real gap, and it is not a graphics problem. you can close it with a flat grey render that gets the forces right, and you can completely fail with a cinematic render that gets them wrong.
the practical consequence is uncomfortable. you cannot verify a simulation by looking at it. a human looking at a render has no way to tell whether the friction coefficient is correct. the only check is to run the same task on real hardware and compare what happened.
which is exactly why real-world recorded episodes stay valuable even in a world full of simulators. the sim is where you get volume. the real data is the only thing that tells you the sim is lying.
beauty is not fidelity. i think that one sentence saves people a lot of wasted compute.
@prismaxai