We’ve shown⁠(opens in a new window) that domain randomization, an existing idea for making detectors trained on simulated images transfer to real images, works well for cluttered scenes. The method is simple: we randomly vary colors, textures, lighting conditions, and camera settings in simulated scenes. The resulting dataset is sufficiently variable to allow a deep neural network trained on it to generalize to reality.

Randomly generated scenes. Each frame contains Spam, often hidden among distractor objects. Our Spam model is sourced from the YCB dataset.

The detector is a neural network based on the VGG16⁠(opens in a new window) architecture that predicts the precise 3-D location of Spam in simulated images. Though it has only been trained on simulated scenes, the resulting network is able to detect Spam in real images, even in the presence of never-before-seen “distractor” items arranged in random configurations.

In the future, we plan to extend this work to detect phishing⁠(opens in a new window) and to defend against adversarial⁠ Spam.