We’ve just released initial results⁠(opens in a new window) in which we teach AI agents to create language by dropping them into a set of simple worlds, giving them the ability to communicate, and then giving them goals that can be best achieved by communicating with other agents. If they achieve a goal, then they get rewarded. We train them using reinforcement learning and, due to careful experiment design, they develop a shared language to help them achieve their goals.

Our approach yields agents that invent a (simple!) language which is grounded⁠(opens in a new window) and compositional⁠(opens in a new window). Grounded means that words in a language are tied to something directly experienced by a speaker in their environment, for example, a speaker forming an association between the word “tree” and images or experiences of trees. Compositional means that speakers can assemble multiple words into a sentence to represent a specific idea, such as getting another agent to go to a specific location.

To train the agents, we represent the experiment as a cooperative—rather than competitive—multi-agent reinforcement learning problem. The agents exist in a two-dimensional world with simple landmarks, and each agent has a goal. Goals can vary from looking at or moving to a specific location, to encouraging a separate agent to move to a location. Each agent can broadcast messages to the group. Every agent’s reward is the sum of the rewards paid out to all agents, encouraging collaboration.

At each time step, our RL agents can take two kinds of actions—(i) environment actions, like moving around or looking at things, and (ii) communication actions, like broadcasting a word to all other agents. (Note that though the agents come up with words that we found to correspond to objects and other agents, as well as actions like “Look at” or “Go to,” to the agents these words are abstract symbols represented by one-hot vector⁠(opens in a new window)—we label these one-hot vectors with English words that capture their meaning for the sake of interpretability.) Before an agent takes an action, it observes the communications from other agents from the previous time step as well as the locations of all entities and objects in the world. It stores that communication in a private recurrent neural network, giving it a memory for the words it hears.