Now, we’ve developed and deployed a new algorithm, one-shot imitation learning⁠(opens in a new window), allowing a human to communicate how to do a new task by performing it in VR⁠(opens in a new window). Given a single demonstration, the robot is able to solve the same task from an arbitrary starting configuration.

Our system can learn a behavior from a single demonstration delivered within a simulator, then reproduce that behavior in different setups in reality.

The system is powered by two neural networks: a vision network and an imitation network.

The vision network ingests an image from the robot’s camera and outputs state representing the positions of the objects. As before⁠(opens in a new window), the vision network is trained with hundreds of thousands of simulated images with different perturbations of lighting, textures, and objects. (The vision system is never trained on a real image.)