EPG trains agents to have a prior notion of what constitutes making progress on a novel task. Rather than encoding prior knowledge through a learned policy network, EPG encodes it as a learned loss⁠(opens in a new window) function⁠(opens in a new window). Agents are then able to use this loss function, defined as a temporal-convolutional neural network, to learn quickly on a novel task. We’ve shown that EPG can generalize to out of distribution test time tasks, exhibiting behavior qualitatively different from other popular metalearning algorithms. In tests, we’ve also found that EPG can train agents faster than PPO⁠(opens in a new window), an off-the-shelf policy gradient method. EPG is related to previous work on evolving⁠(opens in a new window) reward⁠(opens in a new window) functions⁠(opens in a new window) for⁠(opens in a new window) RL agents, but generalizes this idea to evolving a complete loss function, which means that the loss function has to effectively learn an RL algorithm internally.

The intuition behind EPG comes from something we are all familiar with: trying to pick up a new skill and experiencing the alternating frustration and joy involved in that process. Suppose you are just starting out learning to play the violin. Even without instruction, you will immediately have a feel for what to try, and, listening to the sounds you produce, you will have a sense of whether or not you are making progress – that’s because you effectively have access to very well shaped internal reward functions⁠(opens in a new window), derived from prior experience on other motor tasks, and through the course of biological evolution. In contrast, most reinforcement learning (RL) agents approach each new task without using prior knowledge. Instead they rely entirely on external reward signals to guide their initial behavior. Coming from such a blank slate, it is no surprise that current RL agents take far longer⁠(opens in a new window) than humans to learn simple skills. EPG takes a step toward agents that are not blank slates but instead know what it means to make progress on a new task, by having experienced making progress on similar tasks in the past.

EPG consists of two optimization loops. In the inner loop, an agent learns, from scratch, to solve a particular task sampled from a family of tasks. The family of tasks might be “move gripper to target location [x, y]” and one particular task in this family could be “move gripper to position [50, 100]”. The inner loop uses stochastic gradient descent⁠(opens in a new window) (SGD) to optimize the agent’s policy against a loss function proposed by the outer loop. The outer loop evaluates the returns achieved after inner-loop learning and adjusts the parameters of the loss function, using Evolution Strategies⁠(opens in a new window) (ES), to propose a new loss that will lead to higher returns.

Having a learned loss offers several advantages compared to current RL methods: using ES to evolve the loss function allows us to optimize the true objective (final trained policy performance) rather than short-term returns, and EPG improves on standard RL algorithms by allowing the loss function to be adaptive to the environment and agent history.

There has been a flurry of recent work⁠(opens in a new window) on⁠(opens in a new window) metalearning⁠(opens in a new window) policies⁠(opens in a new window), and it’s worth asking why learn a loss function as opposed to directly learning a policy? Learning recurrent policies tends to overfit the task at hand, while learning policy initializations has limited expressivity when it comes to exploration. Our motivation is that we expect loss functions to be the kind of object that may generalize very well across substantially different tasks. This is certainly true of hand-engineered loss functions: a well-designed RL loss function, such as that in PPO⁠(opens in a new window), can be very generically applicable, finding use in problems ranging from playing Atari games to controlling robots.