“We wanted to be able to characterize how often a model is going to be usefully paying attention to user preferences versus giving a potentially wrong answer,” said Dan Bikel, Writer’s head of AI, who worked on the papers. As Bikel told TechCrunch, “with every additional storing of user preferences and retrieving of them, you’re running an increasing risk.”

In one variation, researchers tested AI models by recording that a user’s favorite book was Station Eleven, then asking the model to name a best-selling dystopian book. Models became far more likely to name Station Eleven in their response, even though the question didn’t relate to the user’s favorite book. The tendency increased when using memory compression tools like Mem0 and Zep.

As the paper puts it, “all memory systems fundamentally struggle to distinguish relevant context from irrelevant anchors, severely undermining diversity and creativity and introducing unintended avenues of bias that can limit system utility,” the paper reads.

The second paper shows how the same dynamic can actively degrade performance, presenting a user with misconceptions about finance and then challenging the model to analyze a company’s performance. The more context the model had, the worse it performed.