OpenAI has retracted its own recommendation to use SWE-Bench Pro, the coding benchmark it endorsed just months ago, after an internal audit found roughly 30 per cent of its tasks are broken.
The review follows an earlier finding that SWE-bench Verified, once the industry's default coding eval, had "fundamental design and contamination issues" and could no longer be trusted to reflect real software development capability. OpenAI had pointed developers towards SWE-Bench Pro as the fix.
Now the replacement has problems of its own. An initial automated filter flagged 286 tasks for closer review out of 731 in the public set. A deeper pipeline audit confirmed 200 of these, 27.4 per cent, as genuinely broken, while a parallel human annotation campaign, using five engineers per task, found an even higher rate: 249 tasks, or 34.1 per cent.
The starkest gap between the two review methods came on low-coverage tests, where checks under-specify the requested feature and let incomplete fixes pass. Human reviewers flagged this in 9.4 per cent of the benchmark, more than double the 4.1 per cent picked up by the automated pipeline. Overly strict tests, underspecified prompts and misleading prompts made up the remaining issue categories.
OpenAI says the episode shows evaluation flaws are now easier to catch, since models themselves can audit prompts and test suites at scale, and it is calling for benchmarks built specifically for testing models rather than scraped from real-world pull requests.
