The industry-standard metric for customer service, containment rate, measures the number of customer inquiries that are handled entirely by a chatbot and never reach a human agent. If you increase the containment rate, you’ll theoretically reduce operational expenses in customer service. But the team at Ada saw a problem with this metric.

“Many solutions, including Ada, could easily deliver 80–100% containment rates, but if you actually opened up those conversation transcripts and read the experiences customers had, they were quite poor,” Gozzo said.

By focusing their efforts on how well conversations were actually resolved, the team at Ada knew they could set a new standard in the customer service industry.

Initially using GPT‑4, combined with their historical data, Ada built a new evaluation framework capable of assessing conversations for how well they were resolved automatically. Ada’s system rates each conversation by how well customers receive relevant, accurate, and safe replies—without human intervention. “In our testing, our system achieved 80–90% agreement with a human reading the conversation,” Gozzo said.