Modern neural networks have achieved high accuracies on a wide range of benchmark tasks. However, they remain susceptible to adversarial examples⁠, small but carefully crafted distortions of inputs created by adversaries to fool the networks. For example, the adversarial example with L∞ L_\infty L∞​ distortion below differs from the original image by at most 32 in each RGB pixel value; a human can still classify the changed image, but it is confidently misclassified by a standard neural network.

Sample images (black swan) generated by adversarial attacks with different distortion types. Each distortion is optimized to fool the network.

AI systems deployed in the wild will need to be robust to unforeseen attacks, but most defenses so far have focused on specific known attack types. The field has made progress in hardening models against such attacks; however, robustness against one type of distortion often⁠(opens in a new window) does⁠(opens in a new window) not⁠(opens in a new window) transfer⁠(opens in a new window) to⁠(opens in a new window) robustness against attacks unforeseen by designers of the model. Consequently, evaluating against only a single distortion type can give a false sense of security about a model in the wild which may remain vulnerable to unforeseen attacks such as fake eyeglasses⁠(opens in a new window) and adversarial stickers⁠(opens in a new window).

An example where adversarial robustness does not transfer well. Hardening a model against Distortion A initially increases robustness against both Distortions A and B. However, as we harden further, adversarial robustness is harmed for Distortion B but remains about the same for Distortion A; (A = _L_∞​, B = _L_1​). The accuracy of the model against Distortion A peaks at a hardening level of 8 because that is sufficient to defend against the attack and further hardening hurts clean accuracy; see full paper for details.

We’ve created a three-step method to assess how well a model performs against a new held-out type of distortion. Our method evaluates against diverse unforeseen attacks at a wide range of distortion sizes and compares the results to a strong defense which has knowledge of the distortion type. It also yields a new metric, UAR, which assesses the adversarial robustness of models against unforeseen distortion types.