Rolling out AI in financial services required confidence that the technology will deliver outsized value while meeting the firm’s strict standards for quality and reliability.

Morgan Stanley met this challenge by implementing an evaluation (eval) framework to test every AI use case before deployment. Evals measure how models perform against real-world use cases and guide improvements, with expert feedback, at every step.

To evaluate GPT‑4’s performance against their experts, Morgan Stanley ran summarization evals to test how effectively the model condensed vast amounts of intellectual capital and process-driven content into concise summaries. Advisors and prompt engineers graded AI responses for accuracy and coherence, allowing the team to refine prompts and improve output quality.

The eval framework wasn’t static; it evolved as the team learned. They next introduced translation evals for multilingual clients and worked closely with OpenAI to fine-tune retrieval methods, ensuring AI could handle an ever-expanding document library.