Large language model (LLM) performance and safety in health continue to advance. OpenAI model performance on HealthBench⁠ doubled from GPT‑4o to o3, and frontier models often outperform experts on tasks like diagnostic reasoning⁠(opens in a new window) and clinical summarization⁠(opens in a new window). Yet adoption towards solving real-world patient and clinician challenges remains slow. To realize the potential of LLMs in health, the ecosystem will need to close the model-implementation gap—the chasm between what models can do and how they are used in practice.

To advance research on real-world implementation, OpenAI partnered with Penda Health⁠(opens in a new window), a primary care provider operating in Nairobi, Kenya since 2012, to conduct a novel study of Penda’s LLM-powered clinician copilot. Penda built their copilot, AI Consult, to provide clinicians with LLM-written recommendations at key points during a patient visit. AI Consult acts as a real-time safety net that activates only when there might be an error, keeping clinicians fully in control.

In a study of 39,849 patient visits across 15 clinics, clinicians with AI Consult had a 16% relative reduction in diagnostic errors and a 13% reduction in treatment errors compared to those without.

Today, we are publishing the study findings alongside a closer look at Penda’s successful implementation, offering the ecosystem an early template for the safe and effective use of LLMs to support clinicians.