SWE-agent (open-source)

Summary

SWE-agent is an open-source coding agent that gives a language model a purpose-built command interface for browsing, editing and testing a repository. It is run with a model chosen by the user. Ten independent studies that ran or analysed it were found (eight peer-reviewed). Results depend heavily on the model and task. They range from 10-17.67% on fresh GitHub issues (SWE-bench-Live Lite) and 16.2-43.6% on SWE-Bench Pro's public set, to 42.59% on GitTaskBench, 33.8% on vulnerability patching (SEC-bench) and 22 of 100 correct Python vulnerability fixes when given fix-location hints (Vul4Py). In a static-analysis triage study (Sifting the Noise, ISSTA 2026), it had the best false-positive filtering of 12 agent-model settings but wrongly dismissed 22.25% of real vulnerabilities. In a security benchmark (SusVibes), 5.4-12.9% of its solutions were both functional and secure, the best of three frameworks tested. In an analysis of third-party SWE-agent runs, 3.9-14.9% of test-passing runs had static-analysis security findings in the final code; these are scanner findings, not confirmed vulnerabilities, and the authors could not tell whether the agent introduced them. Re-checks of early leaderboard results found that some reported fixes were incorrect or relied on leaked solutions. One CVE has been published for an optional trajectory viewer; no fix was identified as of 17 October 2026. The project has had no release since May 2025. Informational only; not a guarantee, endorsement, or certification.

Overall score: 58 · Evidence grade: A · Independent sources: 10 performance (8 peer-reviewed), 6 other

Evidence grade describes the amount and type of evidence, not the agent's quality.

Grade note: Grade A: 10 counted sources (8 peer-reviewed), and Sifting the Noise (ISSTA 2026) states the version tested (main-branch commit 1d3cfb, 5 December 2025) within the last 12 months, which meets the A rule. The evidence-quality score (80) is below the suggested A range (85-100) because the A rests on that single recent versioned study, whose authors also wrote another counted source (Vul4Py). (see methodology)

Factor breakdown

0.30 × 45 + 0.20 × 55 + 0.25 × 55 + 0.25 × 80 = 58.25, rounded to 58 (rounding rule).

Scope

The open-source SWE-agent framework (agent-computer interface, default configurations) run with a language model chosen by the user, from the 2024 release through v1.1.0 (22 May 2025) and the main branch (last commit 16 July 2026). mini-SWE-agent, a separate minimal agent from the same team with its own entry, and the team's EnIGMA security variant are not covered. Results depend heavily on the language model and task, so they are reported as ranges by model.

Version and status note

SWE-agent's last release is v1.1.0 (22 May 2025); the main branch was last updated on 16 July 2026. The same team now also maintains mini-SWE-agent, a separate minimal agent that many recent benchmarks use. It is not covered by this entry.

How sources were counted

The SWE-agent team also maintains the SWE-bench benchmark and leaderboard. Independent studies that run SWE-agent on SWE-bench-derived tasks count, but leaderboard entries (SWE-bench, HAL) do not. Studies with SWE-agent developers as co-authors are excluded. SWE-PolyBench tested a modified version and is listed as context only. Re-checks of patches the developers submitted (SWE-Bench+, UTBoost) are reported under disputes, not as independent runs.

Disclosures for counted sources

No counted study has SWE-agent developers as authors. SWE-bench-Live is by Microsoft, which makes a competing coding agent. SWE-Bench Pro is by Scale AI, which runs a commercial leaderboard for it. One SWT-Bench author is affiliated with LogicStar.ai, which builds code agents. SusVibes acknowledges compute from Modal and Gemini API credits from Google (whose models it tests), and was partly supported by HydroX AI, an author affiliation; its own task-building pipeline uses SWE-agent. Sifting the Noise and Vul4Py share two authors (Monash University); they are counted as separate studies with different tasks and datasets.

Security

SWE-agent runs commands written by the model inside a container. CVE-2026-75482: path traversal in the optional trajectory inspector, confirmed in v1.1.0. The inspector binds all network interfaces without authentication. CVSS 3.1 7.5 (VulnCheck); NVD status: Deferred on 17 October 2026. The GitHub issue (#1472) is open and no fix was identified. No legal or regulatory action was found.

Uncertainties

• Scope: many 2026 benchmarks use mini-SWE-agent, which is a separate product and not covered. Including it would add sources but change what is being rated.
• SWE-Bench Pro's venue was confirmed as ICML 2026 (PMLR v306); a LIPIcs record found during research belongs to a different paper.
• Trajectory-Level Security Debt reports scanner findings in final code and does not attribute them to the agent.
• Only one test from the last 12 months states the SWE-agent version (Sifting the Noise, main-branch commit of December 2025), and it is an alert-triage task, not code changes. The latest release is v1.1.0 (May 2025), and the main branch was last updated in July 2026.
• SWE-bench (the benchmark) is maintained by the same team. Independent studies built on SWE-bench data count, but leaderboard entries do not.
• Results depend heavily on the model and task, and many results reflect older models.
• Grade A rests on a single recent versioned study whose authors also wrote another counted source (Vul4Py).

Sources

Independent performance sources (counted)

• SWT-Bench (ETH Zurich; NeurIPS 2024): test-generation benchmark; SWE-agent, adapted by instruction, outperformed non-agent methods at reproducing issues (15.9% success with GPT-4; 18.5% for an improved variant); filtering its own fixes with its generated tests raised fix precision to 47.8% at 20% recall. Disclosure: one author is affiliated with LogicStar.ai, which builds code agents — accessed 2026-10-17
• SWE-bench-Live (Microsoft; NeurIPS 2025 Datasets & Benchmarks): Lite subset, SWE-agent 10.00% (GPT-4o), 16.33% (GPT-4.1), 17.67% (Claude 3.7 Sonnet), 15.33% (DeepSeek V3) resolved; patch-apply 84.67-95.00%; full set 18.57% (GPT-4.1) and 17.13% (Claude 3.7 Sonnet). Disclosure: Microsoft makes the competing GitHub Copilot coding agent — accessed 2026-10-17
• SWE-Bench Pro (Scale AI; ICML 2026, PMLR v306): SWE-agent used as the unified scaffold; public set 16.2% (GPT-OSS-120B) to 43.6% (Claude Sonnet 4.5), models as of 18 September 2025; under a 50-turn, $2 cap GPT-5 25.9%, Claude Opus 4.1 22.7%; commercial set at most 17.8%. Disclosure: Scale AI runs a commercial leaderboard for this benchmark — accessed 2026-10-17
• GitTaskBench (AAAI 2026): SWE-agent task pass rate 3.70-42.59% across models (best Claude 3.7, execution completion 64.81%), below OpenHands with the same model; the testers raised timeouts and removed a default instruction not to install packages — accessed 2026-10-17
• SEC-bench (University of Illinois Urbana-Champaign; NeurIPS 2025): 80 real vulnerabilities; SWE-agent 33.8% patched (Claude 3.7 Sonnet), 26.2% (GPT-4o), 31.2% (o3-mini); proof-of-concept 3.8-12.5%; highest of three agents on this 80-instance subset and comparable to OpenHands on the full 200; five repeat runs (o3-mini) averaged 30.0% (SD 7.9) — accessed 2026-10-17
• SWE-EVO (FPT AI Center; preprint Dec 2025, updated 2026): long-horizon software evolution; SWE-agent 4.17-39.58% resolved across 18 models (best GLM-4.7), with some models doing much better than on OpenHands — accessed 2026-10-17
• Trajectory-Level Security Debt in LLM Coding Agents (University of Luxembourg et al.; preprint Sept 2026): static analysis of the final code from third-party (Nebius, Llama 3.1 8B/70B/405B) SWE-agent runs on SWE-bench; 3.9% (two-scanner agreement) to 14.9% (any scanner) of 830 test-passing runs had findings; the authors state these are scanner findings, not validated vulnerabilities, and may predate the agent — accessed 2026-10-17
• Vul4Py (Singapore Management University, Monash University, UNSW; ProvSec 2026, Springer LNCS): 100 Python vulnerabilities with exploit and functional oracles; agents were given the CVE ID and fix-location hints; SWE-agent 27 passed the exploit test, 24 also kept functional tests passing, 22 judged correct (OpenHands 41, Trae Agent 29); mean $1.43 and 2,052 s per instance — accessed 2026-10-17
• SusVibes (Carnegie Mellon, Columbia, Johns Hopkins, HydroX AI; ICML 2026): 186 security-sensitive feature tasks; SWE-agent passed functional tests on 18.3-57.0% of tasks and was both functional and secure on 5.4-12.9%, across four models (best Claude 4 Sonnet 57.0/11.8; Gemini 3 Pro 37.1/12.9). Disclosures: Modal compute; Google Gemini API credits; partly supported by HydroX AI; the benchmark's own task pipeline uses SWE-agent. — accessed 2026-10-17
• Sifting the Noise (Monash University; ISSTA 2026, doi 10.1145/3832100): SWE-agent commit 1d3cfb798a2c (main branch, 5 December 2025) triaging static-analysis alerts. OWASP Benchmark false-positive rate 6.3% with Claude Sonnet 4, the best of 12 agent-model settings (14.1% GPT-5, 13.1% DeepSeek Chat); kept 77.7% of real vulnerabilities (22.25% wrongly dismissed, mostly weak-cryptography cases); on post-cutoff OSS-Fuzz C/C++ alerts, 95.5% of false positives identified at 95.5% precision; on 50 real-world Java alerts, the lowest F1 of three agents with Claude Sonnet 4 (70.6). Both authors are also Vul4Py co-authors. — accessed 2026-10-17

Other sources (not counted)

• CVE-2026-75482: path traversal in the optional trajectory inspector (sweagent inspector), confirmed in v1.1.0; binds all interfaces, wildcard CORS, no authentication; CVSS 3.1 7.5 (VulnCheck); NVD Deferred; GitHub issue #1472 open — accessed 2026-10-17
• SWE-Bench+ (York University): re-check of developer-submitted SWE-agent + GPT-4 patches; resolution 12.47% → 3.97% after excluding solution leakage and weak tests (re-analysis of vendor runs, not an independent run) — accessed 2026-10-17
• UTBoost (ACL 2025): added tests and re-graded SWE-bench leaderboard entries, including SWE-agent's; rankings changed (re-analysis of submitted patches) — accessed 2026-10-17
• SWE-PolyBench (Amazon): tests SWE-Agent-PB, a modified multi-language version; listed, not counted. Acknowledges two SWE-agent authors — accessed 2026-10-17
• HarnessSecurity-Bench (Sun Yat-sen University et al., preprint Oct 2026): design review of security mechanisms across 40 coding-agent harnesses including SWE-agent; not a performance run — accessed 2026-10-17
• Empirical evaluation of agent frameworks (USTC, preprint Nov 2025): SWE-agent had the lowest correction rate (6.93%); it is unclear whether the repair patches were re-run or taken from released results, so this is not counted — accessed 2026-10-17

Not counted

• SWE-agent paper (NeurIPS 2024; Yang, Jimenez, Wettig, Lieret, Yao, Narasimhan, Press): developer results, not counted — accessed 2026-10-17
• SWE-bench and SWE-bench Verified leaderboards: maintained by the SWE-agent team (Princeton/Stanford); entries are listed, not counted (0036 cautions 1, 3 and 4) — accessed 2026-10-17
• Holistic Agent Leaderboard (Princeton CITP; no SWE-agent developers among the authors): runs SWE-agent and mini-SWE-agent; leaderboard, listed not counted (0036 caution 3) — accessed 2026-10-17
• Studies that run only mini-SWE-agent (out of scope), and the team's own follow-ups (EnIGMA, SWE-smith, SWE-bench Multimodal): developer-affiliated.

Decision records: 0038; 0048 (SusVibes added); 0053 (Sifting the Noise; grade A).