This release ships with eight robotics environments for Gym⁠(opens in a new window) that use the MuJoCo⁠(opens in a new window) physics simulator. The environments are:

All of the new tasks have the concept of a “goal”, for example the desired position of the puck in the slide task or the desired orientation of a block in the hand block manipulation task. All environments by default use a sparse reward of -1 if the desired goal was not yet achieved and 0 if it was achieved (within some tolerance). This is in contrast to the shaped rewards used in the old set of Gym continuous control problems, for example Walker2d-v2⁠(opens in a new window) with its shaped reward⁠(opens in a new window).

We also include a variant with dense rewards for each environment. However, we believe that sparse rewards are more realistic in robotics applications and we encourage everyone to use the sparse reward variant instead.

Alongside these new robotics environments, we’re also releasing code⁠(opens in a new window) for Hindsight Experience Replay⁠(opens in a new window) (or HER for short), a reinforcement learning algorithm that can learn from failure. Our results show that HER can learn successful policies on most of the new robotics problems from only sparse rewards. Below, we also show some potential directions for future research that could further improve the performance of the HER algorithm on these tasks.