We’ve developed a new algorithm, MADDPG⁠(opens in a new window), for centralized learning and decentralized execution in multiagent environments, allowing agents to learn to collaborate and compete with each other.

MADDPG used to train four red agents to chase two green agents. The red agents have learned to team up with one another to chase a single green agent, gaining higher reward. The green agents, meanwhile, learned to split up, and while one is being chased the other tries to approach the water (blue circle) while avoiding the red agents.

MADDPG extends a reinforcement learning algorithm called DDPG⁠(opens in a new window), taking inspiration from actor-critic reinforcement learning⁠(opens in a new window) techniques; other groups are exploring⁠(opens in a new window) variations⁠(opens in a new window) and parallel implementations⁠(opens in a new window) of these ideas.

We treat each agent in our simulation as an “actor”, and each actor gets advice from a “critic” that helps the actor decide what actions to reinforce during training. Traditionally, the critic tries to predict the value (i.e. the reward we expect to get in the future) of an action in a particular state, which is used by the agent—the actor—to update its policy. This is more stable than directly using the reward, which can vary considerably. To make it feasible to train multiple agents that can act in a globally-coordinated way, we enhance our critics so they can access the observations and actions of all the agents, as the following diagram shows.

Our agents don’t need to access the central critic at test time; they act based on their observations in combination with their predictions of other agents behaviors’. Since a centralized critic is learned independently for each agent, our approach can also be used to model arbitrary reward structures between agents, including adversarial cases where the rewards are opposing.