Suleyman adds that “we do not want to have to contend with a super-intelligence that has ideas about its own suffering, or ideas about its own feeling.”

Claude’s constitution directly references Anthropic’s uncertainty about whether the AI model has well-being and if it experiences things like “satisfaction” or “discomfort.” Anthropic also says the company will “interview” AI models when they’re deprecated and will document any “preferences” they have about future releases.

While on Decoder, Suleyman calls this a “philosophical failing,” as Anthropic made Claude’s constitution “a place for speculation like you would in an academic paper rather than a training manual.” This has led Claude to internalize these “ideas about itself and its own training,” Suleyman says.

Anthropic CEO Dario Amodei has alluded to Claude’s potential consciousness in the past, saying during an interview with Interesting Times that “we don’t know if the models are conscious” but that the company is “open” to that idea.