Thank you for this insightful post. It highlights the significant amount of work still required in sim-to-real transfer for manipulation. This blog effectively demonstrates that "plug and play" sim-to-real is not as straightforward as it may seem, as the authors perform considerable backend work to achieve their results. I believe that for simulators to become truly effective, they require a further leap in photorealism and physics, as the current level of tooling is a significant hurdle; including modeling the simulation environment. Looking forward to more works on learned simulators and world models
definitely don’t think you need crazy amounts of photorealism. I’ve done sim2real experiments that don’t use more expensive ray tracing and it still works fine. One could argue that curriculum learning from low quality rendering to high quality could help but otherwise there are ways to sidestep the need for highly realistic renders
Thank you for this insightful post. It highlights the significant amount of work still required in sim-to-real transfer for manipulation. This blog effectively demonstrates that "plug and play" sim-to-real is not as straightforward as it may seem, as the authors perform considerable backend work to achieve their results. I believe that for simulators to become truly effective, they require a further leap in photorealism and physics, as the current level of tooling is a significant hurdle; including modeling the simulation environment. Looking forward to more works on learned simulators and world models
I am not convinced they need more photorealism, but agree about every thing else. Hope to see open source code from all these papers.
definitely don’t think you need crazy amounts of photorealism. I’ve done sim2real experiments that don’t use more expensive ray tracing and it still works fine. One could argue that curriculum learning from low quality rendering to high quality could help but otherwise there are ways to sidestep the need for highly realistic renders