Sora is a diffusion model, which generates a video by starting off with a base video that looks like static noise and gradually transforms it by removing the noise over many steps. By giving the model foresight of many frames at a time, we’ve solved a challenging problem of making sure a subject stays the same even when it goes out of view temporarily. Similar to GPT models, Sora uses a transformer architecture, unlocking superior scaling performance.

Sora uses the recaptioning technique from DALL·E 3, which involves generating highly descriptive captions for the visual training data. As a result, the model is able to follow the user’s text instructions in the generated video more faithfully.

In addition to being able to generate a video solely from text instructions, the model is able to take an existing still image and generate a video from it, animating the image’s contents with accuracy and attention to small detail. The model can also take an existing video and extend it or fill in missing frames⁠. Sora serves as a foundation for models that can understand and simulate the real world, a capability we believe will be an important milestone for achieving AGI.

Sora’s capabilities may also introduce novel risks, such as the potential for misuse of likeness or the generation of misleading or explicit video content. In order to safely deploy Sora in a product, we built on learnings from safety work for DALL·E’s deployment in ChatGPT and the API and safety mitigations for other OpenAI products such as ChatGPT. This system card outlines the resulting mitigation stack, external red teaming efforts, evaluations, and ongoing research to refine these safeguards further.

As described in our technical report⁠1 from February 2024, Sora takes inspiration from large language models which acquire generalist capabilities by training on internet-scale data. The success of the LLM paradigm is enabled in part by the use of tokens that elegantly unify diverse modalities of text—code, math and various natural languages. With Sora, we considered how generative models of visual data can inherit such benefits. Whereas LLMs have text tokens, Sora has visual patches. Patches have previously been shown to be an effective representation for models of visual data. We found that patches are a highly-scalable and effective representation for training generative models on diverse types of videos and images. At a high level, we turn videos into patches by first compressing videos into a lower-dimensional latent space, and subsequently decomposing the representation into spacetime patches.