At my last job, I’d moved from a backend role into a full-stack position, only to find a distaste for frontend work and UX design. I wanted to move back to a role closer to backend systems and missed the interaction with Linux environments I’d enjoyed in academia. OpenAI offered the change in work I was looking for and then some; you’d be hard-pressed to find a better fit for what I was looking for than working on OpenAI’s supercomputing clusters.

Exploratory AI workflows are inherently fast-paced; researchers want to be able to take a preprint off of arXiv and test out new approaches without being encumbered by the platform they’re launching their code on. They are also incredibly complicated, with researchers behaving much like mathematicians—relying on the intuition they’ve built over their careers to design a solution in tackling whatever problem has caught their eye this week. The fact these runtimes are executing on some of the world’s largest supercomputers adds yet another layer of complexity, and handling that penultimate layer is where my team gets involved. We work to preempt research needs before they block progress and, failing that, we work with research teams to identify bottlenecks and implement workarounds as quickly as possible.

The sheer scale we operate at is, frankly, astonishing. Third-party hardware vendors routinely confide that we’re encountering issues they’ve never previously seen. Often this is simply because our installations have more hardware shoved into a single contiguous supercomputer than their other clients, although occasionally it’s a consequence of our performance expectations. The synchronized nature of most model training approaches results in a configuration where the entire cluster effectively runs at the speed of the slowest node.

Our most prominent models are trained on billion-dollar supercomputers, and as a result, we end up chasing down performance degradations that most others would ignore. It’s exciting to see something like a one-line change hit the mainline kernel, knowing that it’ll save ~6 days of compute across our fleet per week, or see a line item on a new driver release, knowing that it was one of our discoveries that resulted in the now-upstreamed fix.