OpenAI finds reliability issues in SWE-Bench Pro coding benchmark
Which summary reads better? Pick one — models revealed after.Both summaries are AI-generated.
SWE-bench Verified reduces the noise and false negatives in the original SWE-bench dataset by manually filtering out underspecified instructions, incorrect unit tests, and overly rigid evaluation criteria. For production agent teams, this means your patch-generation pipelines are no longer being penalized by broken benchmark tests, allowing you to trust that a higher score directly correlates with better real-world code generation rather than overfitting to noisy evaluations. You should migrate your engineering agents' regression testing to the Verified subset immediately to get a true signal on code quality.
The SWE-Bench Pro coding benchmark contains inconsistencies and noise, compromising its reliability for evaluating AI coding models. This undermines the ability to accurately assess model performance in real-world coding tasks, requiring engineers to seek more robust evaluation frameworks.