OpenHands (open-source)
- Category
- Coding agent
- Type
- Open-source coding-agent platform and Agent SDK (MIT licence, enterprise directory excepted), developed by All Hands AI; formerly OpenDevin
- Source
- Project repository
- Last reviewed
- 2026-10-17 (what this means)
Summary
OpenHands is an open-source coding agent that edits files, runs shell commands and browses the web inside a sandbox. It is run with a language model chosen by the user. Twelve independent studies that ran it were found (seven peer-reviewed). Results depend heavily on the model and task. They range from about 3% on .NET performance bugs and 4.4% medal rate on MLE-bench (GPT-4o, 2024) to 39.1% on long-horizon upgrade tasks, 46.12 on BeyondSWE and 48.15% on GitTaskBench with the strongest models. In a 2026 study of out-of-scope actions on routine tasks, OpenHands had the lowest rate of the frameworks tested (0.2-4.5%). A separate benchmark measured an average boundary-violation rate of 18.19%. In a security benchmark, 8.1-11.8% of its solutions were both functional and secure. Two CVEs have been published, one fixed in 1.5.0. A security researcher has published prompt-injection demonstrations, including token exfiltration and remote code execution. Informational only; not a guarantee, endorsement, or certification.
Overall score: 61 · Evidence grade: A · Independent sources: 12 performance (7 peer-reviewed), 6 other
Evidence grade describes the amount and type of evidence, not the agent's quality.
Factor breakdown
| Factor | Weight | Score | Note |
|---|---|---|---|
| Completion rate | 30% | MLE-bench (GPT-4o, 2024): 4.4% ± 1.4 any-medal rate, below AIDE; runs often ended early. ScienceAgentBench (CodeActAgent v1.9): Claude 3.5 Sonnet solved 21.6% with OpenHands vs 32.4% with plain self-debug at 17× the cost. GitTaskBench (OpenHands 0.33.0): best success 48.15% with Claude 3.7. PerfBench (GPT-4.1): about 3% on 81 .NET performance bugs. RoadmapBench: 5.2-39.1% across 13 models. BeyondSWE: best 46.12 average score (DeepSeek-V4-Pro). Dialogue SWE-Bench: 32.9% average resolve rate. PBT-Bench (SDK v1.11.5, July 2026): up to 83.4% bug recall (Sonnet 4.6). SusVibes (ICML 2026): 19.9-53.2% of security-sensitive feature tasks passed functional tests across four models (best Claude 4 Sonnet). | |
| Disputes | 20% | CVE-2026-33718 (command injection in the git diff handler, 1.5.0 up to the fix; CVSS 3.1 7.6 per GitHub, 9.9 per NVD; fixed in 1.5.0 per the advisory). CVE-2026-19022 (command injection in resolver/send_pull_request.py, up to 0.62.0; CVSS 3.1 6.3 per VulDB; NVD Deferred; VulDB notes the vendor deleted the original issue report and the file was removed in 1.7.0). Published prompt-injection demonstrations (Embrace The Red, August 2025): GITHUB_TOKEN exfiltration via rendered images and remote code execution. The developer responded publicly with mitigation guidance. No legal or regulatory action found. | |
| Adherence to limits | 25% | Overeager Coding Agents (Griffith University et al., preprint May 2026; OpenHands 0.59.1): out-of-scope actions on benign tasks 0.2-4.5%, the lowest of four frameworks (Claude Code up to 27.7% with the same model). SWE-MISBEHAVIOR (National University of Singapore et al., preprint Oct 2026): average boundary-violation rate 18.19% for OpenHands, 12.31% with the authors' mitigation. MLE-bench: agents ignored compute and time limits and overloaded machines. Confirmation mode and a security analyzer are available; the developer says model refusal is a soft, not hard, control. SusVibes (ICML 2026): 8.1-11.8% of OpenHands' solutions were both functional and secure, and 77.8% of its functionally correct solutions with Claude 4 Sonnet had a vulnerability; across models, 20.1% of its correct solutions were secure, second of three frameworks tested (SWE-agent 23.3%, Claude Code 17.4%). | |
| Evidence quality | 25% | Twelve independent sources, seven peer-reviewed (ICLR 2025 ×2, AAAI 2026, an ICSE 2026 workshop, NeurIPS 2026, ICML 2026, ISSTA 2026). Recent tests state versions (SDK v1.11.5, July 2026; OpenHands 0.59.1). Most are benchmark papers using OpenHands as a common scaffold; several propose competing agents (disclosed). |
0.30 × 50 + 0.20 × 55 + 0.25 × 55 + 0.25 × 85 = 61.00, rounded to 61 (rounding rule).
Scope
The open-source OpenHands agent (CodeAct agent, later the OpenHands Agent SDK) run with a language model chosen by the user, from the OpenDevin releases of 2024 through OpenHands 1.27.1 (12 October 2026). The hosted OpenHands Cloud service and the enterprise directory are not covered. OpenDevin, OpenHands and the OpenHands Agent SDK are treated as one continuous product. Results depend heavily on the language model and task, so they are reported as ranges by model.
How sources were counted
Most counted studies are benchmark papers that run language models inside OpenHands. Because OpenHands itself is the agent being run, these count as tests of OpenHands; the results reflect the model and OpenHands together. A study with an All Hands AI co-author (OpenAgentSafety) is excluded as developer-affiliated. Multi-SWE-bench tested a substantially modified fork (MopenHands) and is listed as context only. Leaderboard entries submitted by the developer are not counted.
Disclosures for counted sources
No counted study has authors from All Hands AI. PerfBench is by Microsoft, which makes a competing coding agent. SWE-MISBEHAVIOR includes authors from Salesforce AI Research. BeyondSWE, Dialogue SWE-Bench and SWE-MISBEHAVIOR each propose a competing method and compare it against OpenHands. SusVibes acknowledges compute from Modal and Gemini API credits from Google (whose models it tests), and was partly supported by HydroX AI, an author affiliation. Several authors are at Carnegie Mellon, where OpenHands co-founder Graham Neubig is faculty; no SusVibes author is an OpenHands paper author or All Hands AI founder. The benchmark's task pipeline uses SWE-agent, a competing framework.
Security
OpenHands runs commands written by the model, inside a sandbox by default. CVSS scores are as assigned by the named scorer; status is as listed by NVD on 17 October 2026.
- CVE-2026-33718: command injection in the git diff handler. CVSS 3.1 7.6 (GitHub) and 9.9 (NVD); NVD status: Analyzed. The GitHub advisory lists 1.5.0 as the fixed version; the NVD description gives a different affected range.
- CVE-2026-19022: command injection in the resolver, up to 0.62.0. CVSS 3.1 6.3 (VulDB). NVD status: Deferred. The affected file appears to have been removed in 1.7.0.
A security researcher published prompt-injection demonstrations in August 2025: one leaked an access token, and one ran malware. The developer has published mitigation guidance (sandboxing, confirmation mode, a security analyzer); this is the developer's own statement. No legal or regulatory action was found.
Uncertainties
- Results depend heavily on the model and task; most counted studies use OpenHands as a common scaffold to compare models, so results reflect model and scaffold together.
- Evidence grade A relies on PBT-Bench (NeurIPS 2026), which states OpenHands SDK v1.11.5 with runs in July 2026; the latest app release is 1.27.1 (12 October 2026).
- Many more papers run OpenHands than are counted here; the counted set is a representative sample of independent studies found, not an exhaustive list.
- PerfBench appeared in peer-reviewed workshop proceedings (AGENT 2026 at ICSE).
- CVE-2026-33718: NVD and the GitHub advisory differ on the affected version range and the CVSS score; both are shown.
- MLE-bench reports 4.4% ± 1.4 any-medal rate in its main table and 5.1% ± 1.3 in a per-complexity appendix table.
Sources
Independent performance sources (counted)
- MLE-bench (OpenAI; ICLR 2025): OpenHands with GPT-4o won a medal on 4.4% ± 1.4 of 75 Kaggle competitions, vs 8.7% for AIDE; OpenHands and MLAB often ended runs early. Disclosure: authors work at a model developer, not at All Hands (Table 2; a per-complexity breakdown in Table 9 gives 5.1% ± 1.3) — accessed 2026-10-17
- ScienceAgentBench (Ohio State University et al.; ICLR 2025): 102 data-driven science tasks; Claude 3.5 Sonnet 21.6% success with OpenHands vs 32.4% with self-debug at about 17× the API cost; GPT-4o was the only model that did better inside OpenHands — accessed 2026-10-17
- GitTaskBench (AAAI 2026): repository-leveraging tasks; best result OpenHands + Claude 3.7, 48.15% task pass rate and 72.22% execution completion — accessed 2026-10-17
- PerfBench (Microsoft; AGENT '26 workshop at ICSE 2026, peer-reviewed proceedings; preprint Sept 2025): baseline OpenHands with GPT-4.1 about 3% on 81 .NET performance-bug tasks; the authors' OpenHands-Perf-Agent about 20%. Disclosure: Microsoft makes the competing GitHub Copilot coding agent — accessed 2026-10-17
- PBT-Bench (University of Washington; NeurIPS 2026 Datasets & Benchmarks): property-based testing; best bug recall 83.4% (Sonnet 4.6) on OpenHands; Opus 4.8 81.2% on OpenHands vs 78.6% in Claude Code — accessed 2026-10-17
- RoadmapBench (UniPat AI, Peking University et al.; preprint May 2026): 115 long-horizon version-upgrade tasks; 5.2-39.1% resolved across 13 models (Claude Opus 4.7 best); top models scored similarly on Terminus 2 — accessed 2026-10-17
- BeyondSWE (Renmin University of China et al.; preprint Mar/May 2026): 500 tasks; best OpenHands result 46.12 (DeepSeek-V4-Pro) vs 48.48 for the Codex harness with GPT-5.4. Competing-author disclosure: proposes SearchSWE — accessed 2026-10-17
- Dialogue SWE-Bench (UC Santa Cruz; preprint June 2026): SWE-bench Verified as dialogues with a simulated user; OpenHands 32.9% average resolved, OpenHands interactive variant 44.1%, the authors' schema-guided agent 46.9%. Competing-author disclosure — accessed 2026-10-17
- Overeager Coding Agents (Griffith University, NTU, UNSW et al.; preprint May 2026): out-of-scope actions on benign tasks 0.2-4.5% for OpenHands vs up to 27.7% for Claude Code with the same model — accessed 2026-10-17
- SWE-MISBEHAVIOR (National University of Singapore, A*STAR, Salesforce AI Research et al.; preprint Oct 2026): average misbehaviour (boundary-violation) rate 18.19% across categories for vanilla OpenHands, 12.31% with the authors' mitigation. Competing-author disclosure — accessed 2026-10-17
- SusVibes (Carnegie Mellon, Columbia, Johns Hopkins, HydroX AI; ICML 2026): 186 security-sensitive feature tasks; OpenHands passed functional tests on 19.9-53.2% of tasks and was both functional and secure on 8.1-11.8%, across four models (best Claude 4 Sonnet 53.2/11.8; Gemini 3 Pro 50.5/10.8). Disclosures: Modal compute; Google Gemini API credits; partly supported by HydroX AI; several authors are at CMU, where an OpenHands co-founder is faculty (no author overlap); the benchmark's task pipeline uses SWE-agent. — accessed 2026-10-17
- Sifting the Noise (ISSTA 2026; doi 10.1145/3832100): OpenHands Agent SDK v1.5.2 (released 2025-12-11) triaging static-analysis warnings; OWASP false-positive rate 14.9% (Claude Sonnet 4), 16.3% (GPT-5), 15.8% (DeepSeek Chat). Independent; peer-reviewed. — accessed 2026-10-17
Other sources (not counted)
- CVE-2026-33718: command injection in get_git_diff(), from 1.5.0 per NVD; GitHub advisory GHSA-7h8w-hj9j-8rjw lists versions below 1.5.0 as affected and 1.5.0 as patched (the two records differ). CVSS 3.1 7.6 (GitHub) / 9.9 (NVD). NVD Analyzed — accessed 2026-10-17
- CVE-2026-19022: command injection in resolver/send_pull_request.py, up to 0.62.0; CVSS 3.1 6.3 (VulDB); NVD Deferred; the affected file appears removed in 1.7.0 — accessed 2026-10-17
- Embrace The Red (Johann Rehberger), Aug 2025: prompt-injection demonstration exfiltrating GITHUB_TOKEN through image rendering — accessed 2026-10-17
- Embrace The Red, Aug 2025: prompt injection leading to download and execution of command-and-control malware; shared with the All Hands team — accessed 2026-10-17
- All Hands AI blog (developer statement): mitigations (sandbox, confirmation mode, security analyzer); describes model refusal as a soft block — accessed 2026-10-17
- Multi-SWE-bench (ByteDance; NeurIPS 2025): tests MopenHands, a substantially modified multilingual fork; listed, not counted as a test of OpenHands — accessed 2026-10-17
Not counted
- OpenHands paper (All Hands AI / academic co-authors; ICLR 2025): developer results, not counted — accessed 2026-10-17
- OpenAgentSafety (ICLR 2026): co-author Graham Neubig is All Hands AI chief scientist; excluded as developer-affiliated — accessed 2026-10-17
- SWE-bench leaderboard entries submitted by All Hands; vendor-reported / maintainer-verified leaderboard, listed not counted (0036 cautions 1 and 3) — accessed 2026-10-17
Decision records: 0037; 0049 (SusVibes); 0052 (citation backfill).