ML//benchmark//SWE-Bench

SWE-Bench is a benchmark built from real GitHub issues in popular Python repositories: the model must write a patch that passes the test suite. It tests agent capabilities, because solving a real issue takes planning, exploring the repository, debugging and multi-step tool use, well beyond generating code.

SWE-Bench is a benchmark built from real GitHub issues in popular Python repositories: the model must write a patch that passes the test suite. It tests agent capabilities, because solving a real issue takes planning, exploring the repository, debugging and multi-step tool use, well beyond generating code.

SWE-Bench Verified is its human-validated subset, adopted by Anthropic, Devin and OpenAI as the de facto agent benchmark.