Dictionary term
SWE-Bench
A family of benchmarks (SWE-bench, Lite, Verified, Multilingual, Multimodal, and the separate SWE-bench Pro) that test whether an AI system can resolve real GitHub issues from open-source projects by producing a patch that passes the project's tests.
In plain English
SWE-bench is a test that checks whether an AI can fix real bugs from real software projects. Scores are useful, but some versions of the test have known flaws.
Analogy
Like a report card or coding exam for AI programmers.
Why it matters
It is one of the most cited signals of coding ability, but treat any single score with care. OpenAI stopped reporting SWE-bench Verified, calling it increasingly contaminated and recommending SWE-bench Pro, and Epoch AI rates Verified as flawed. Compare models on the same variant and harness, and test them on your own codebase before choosing one.
In practice
Two vendors each quote a high SWE-bench score, but one used Verified and the other used Pro, so the numbers can't be compared. A team runs both models against twenty of its own past bug tickets instead.
Related terms
Sources
Browse all terms →Last verified