OpenAI analyzed SWE-Bench Pro, a widely used coding benchmark, and found reliability and accuracy issues. These problems could affect how AI coding models are evaluated. The findings highlight the need for better benchmarks in AI coding assessments.
Source preview
Server-side extracted preview paragraphs from the original source.
A new analysis from OpenAI reveals issues in SWE-Bench Pro, a popular coding benchmark, raising concerns about reliability and accuracy in evaluating AI models.
Through a detailed audit, we find widespread task issues in SWE-Bench Pro and estimate that ~30% of the tasks are broken.