Recognizing this gap, Dr. Emily Oster, a Brown economist and author, set out to change the narrative. “Zelma makes it possible for parents and policymakers alike to use plain language to get instant, tailored educational insights on what matters most to them,” Dr. Oster explained. How did Zelma accomplish this? Dr. Oster’s team of student researchers at Brown University spent a year gathering the data and meticulously cleaning it into one uniform format. Zelma then worked with Novy⁠(opens in a new window) to bring the data to life using OpenAI’s API.

Novy leveraged function calling⁠(opens in a new window) to tell GPT‑4 which visuals and fields to choose from in displaying the data. For Zelma’s "Ask a Question" view, they fine-tuned⁠(opens in a new window) a model to create a data-aware type-ahead that suggests questions with available data. And they embedded⁠ and stored known-good example graphs in a vector database to improve accuracy on difficult edge cases.

The biggest challenge was trying to predict how people would phrase their questions, and keeping the prompts within Zelma’s scope of knowledge. “We addressed this through intentional design choices in Zelma’s user experience that nudge people to ask questions Zelma can answer,” Dr. Oster explained. Key features include:

Now, you can instantly shortlist the top 5 highest performing non-charter school districts in math in Minnesota,⁠(opens in a new window) and find that they all score around a 77 percent proficiency rate. And a simple prompt can surface racial disparities in ELA achievement in California over time,⁠(opens in a new window) while also pointing out an assessment change in 2015 that makes test results from 2015 onward not comparable with prior years.