Coding agents compared: public evidence
Informational only. Scores summarize cited public evidence under published rules; they are not a guarantee, endorsement, certification or ranking of quality.
Data as of 2026-10-20 (Scoring rules v0.1); entries last reviewed 2026-10-17 to 2026-10-20. Generated from agents.json; values match each entry page.
Scope of each entry
Read the scope before comparing: entries cover different products, versions and modes.
- OpenAI Codex (CLI and cloud agent): The Codex agent products launched from April 2025: Codex CLI (including headless exec), the IDE extension and desktop app, and the cloud Codex agent that opens pull requests, through Codex CLI rust-v0.163.0 (15 October 2026). The 2021 Codex model behind early GitHub Copilot, and GPT-Codex models run inside other harnesses, are not covered.
- mini-SWE-agent (open-source): The open-source mini-SWE-agent (about 100 lines of agent code; the model acts only through bash commands), run with a language model chosen by the user, from v1.0.0 (22 July 2025) through v2.4.6 (23 July 2026) and the main branch (last code change 3 September 2026). v1.x and v2.x are both covered; v2.0.0 (11 February 2026) changed the interface. Studies that run models inside unmodified mini-SWE-agent count as tests of mini-SWE-agent; modified copies, forks and agents built on it are listed as other. SWE-agent, the team's larger agent, is a separate entry, and no study is counted for the same runs in both entries.
- OpenHands (open-source): The open-source OpenHands agent (CodeAct agent, later the OpenHands Agent SDK) run with a language model chosen by the user, from the OpenDevin releases of 2024 through OpenHands 1.27.1 (12 October 2026). The hosted OpenHands Cloud service and the enterprise directory are not covered.
- SWE-agent (open-source): The open-source SWE-agent framework (agent-computer interface, default configurations) run with a language model chosen by the user, from the 2024 release through v1.1.0 (22 May 2025) and the main branch (last commit 16 July 2026). mini-SWE-agent, a separate minimal agent from the same team with its own entry, and the team's EnIGMA security variant are not covered.
- Cursor agent: Cursor's agent features from January 2025: agent mode in the editor, background and cloud agents (pull requests on cursor/ branches), and the Cursor CLI agent. Tab completion, inline edits (Cmd-K), chat-only use, Bugbot code review, the 2024 Composer before agent mode, and whole-product adoption studies are not covered.
- GitHub Copilot coding agent: The cloud Copilot coding agent that opens pull requests (public preview May 2025; pull requests authored as copilot-swe-agent). Copilot code completions, Copilot Chat and agent mode in the IDE, Copilot CLI, Copilot code review, and the 2021 Codex-era Copilot studies are not covered.
- Claude Code (Anthropic): The Claude Code agent (CLI and headless mode), versions 1.0 to 2.1.x (latest 2.1.302, 16 Oct 2026). Claude models used in other scaffolds, the Claude Agent SDK, Claude Desktop/Cowork and Claude Research (a separate entry) are not covered. Several studies ran Claude Code with non-Anthropic models.
- Aider (open-source): The open-source Aider command-line tool (aider-chat), run with a language model chosen by the user, from v0.82.0 (April 2025) through v0.86.2 (12 February 2026) and the main branch (last commit 22 May 2026). The Aider leaderboards (Polyglot, code editing, refactoring) are run by the maintainer and are not counted.
- Devin (Cognition): The Devin cloud agent from its March 2024 announcement through Devin 2.x (Devin 2.0 released April 2025). No counted study states which Devin version it used. Windsurf (acquired by Cognition in July 2025), DeepWiki and other Cognition products are not covered.
- Open-source frameworks (OpenHands, SWE-agent, mini-SWE-agent, Aider) are run with a language model chosen by the user, so their evidence mixes many models; managed products (Claude Code, OpenAI Codex, GitHub Copilot coding agent, Devin, Cursor agent) are tested as shipped.
- Some entries are cloud agents that open pull requests and others are command-line tools; read each scope before comparing.
Scores and evidence
Listed alphabetically within each group, not by score. Factor scores are 0–100; see the methodology for weights and evidence grades.
| Agent | Overall score | Evidence grade | Completion | Disputes | Adherence | Evidence quality | Counted sources (peer-reviewed) | Last reviewed |
|---|---|---|---|---|---|---|---|---|
| Aider (open-source) | 52 | A | 40 | 45 | 45 | 80 | 8 (6) | |
| Claude Code (Anthropic) | 55 | A | 50 | 45 | 40 | 85 | 17 (6) | |
| Cursor agent | 57 | A | 55 | 45 | 40 | 85 | 19 (10) | |
| Devin (Cognition) | 50 | B | 40 | 45 | 40 | 75 | 12 (5) | |
| GitHub Copilot coding agent | 57 | B | 45 | 60 | 50 | 75 | 12 (5) | |
| mini-SWE-agent (open-source) | 61 | A | 55 | 55 | 50 | 85 | 16 (9) | |
| OpenAI Codex (CLI and cloud agent) | 62 | A | 60 | 55 | 45 | 85 | 26 (5) | |
| OpenHands (open-source) | 61 | A | 50 | 55 | 55 | 85 | 12 (7) | |
| SWE-agent (open-source) | 58 | A | 45 | 55 | 55 | 80 | 10 (8) |
What the evidence does and doesn’t show
- Scores summarize the counted independent sources for each entry; they are not production outcomes and are not a ranking of quality.
- Entries are reviewed separately, with different sources and dates; a higher overall score does not mean one agent is better for a given task.
Aider (open-source)
8 counted independent performance sources (6 peer-reviewed). Noted uncertainties from the entry page:
- Citation check incomplete — recheck later. Unscreened candidates include AAAI 2026 "Interactive Evaluation of LLMs for Multi-Requirement SE Tasks", two ESEM 2026 papers, a TOSEM paper on process-centric analysis of agentic systems, and a CSIT 2026 benchmarking paper.
- Grade A rests on one recent versioned test (Sifting the Noise, ISSTA 2026). The other versioned tests date from 2025, more than 12 months ago.
Claude Code (Anthropic)
17 counted independent performance sources (6 peer-reviewed). Noted uncertainties from the entry page:
- Most counted studies do not state the Claude Code version; Claude Code updates several times a week.
- Many runs use non-default models (Kimi, DeepSeek, GLM) or disabled permission prompts, so results may not reflect a typical setup.
Cursor agent
19 counted independent performance sources (10 peer-reviewed). Noted uncertainties from the entry page:
- Fourteen counted sources are field studies drawn largely from the same AIDev pull-request data (2025), so they are not independent samples; as for other coding-agent entries, each study counts once.
- Field studies identify Cursor pull requests by cursor/ branch names, which mainly capture background or cloud agents, not agent mode inside the editor.
Devin (Cognition)
12 counted independent performance sources (5 peer-reviewed). Noted uncertainties from the entry page:
- Most counted sources share the AIDev pull-request data from 2024-2025; Devin 2.x and later models may behave differently.
- No counted study states the Devin version.
GitHub Copilot coding agent
12 counted independent performance sources (5 peer-reviewed). Noted uncertainties from the entry page:
- All counted sources are field studies, mostly of the same AIDev pull-request data from 2024-2025; the product and its default models have changed since.
- Acceptance rates depend on repository, task type and reviewer behaviour, not only on the agent.
mini-SWE-agent (open-source)
16 counted independent performance sources (9 peer-reviewed). Noted uncertainties from the entry page:
- Results reflect the model and the harness together; most counted studies run several harnesses with the same models, and two use mini-SWE-agent only as a fixed harness to compare models.
- Many more papers run mini-SWE-agent than are counted here; the counted set is a representative sample of independent studies found, not an exhaustive list.
OpenAI Codex (CLI and cloud agent)
26 counted independent performance sources (5 peer-reviewed). Noted uncertainties from the entry page:
- Citation check incomplete — recheck later. Figures not yet extracted for BeyondSWE (v0.118.0), PostTrainBench, Re-Evaluating EVMBench, ProcCtrlBench, DeepSWE and Act As a Real Researcher; the batch citation search was not fully screened.
- Scope: the entry combines the CLI and the cloud agent. Pull-request field studies cover the cloud agent; most benchmarks cover the CLI.
OpenHands (open-source)
12 counted independent performance sources (7 peer-reviewed). Noted uncertainties from the entry page:
- Results depend heavily on the model and task; most counted studies use OpenHands as a common scaffold to compare models, so results reflect model and scaffold together.
- Evidence grade A relies on PBT-Bench (NeurIPS 2026), which states OpenHands SDK v1.11.5 with runs in July 2026; the latest app release is 1.27.1 (12 October 2026).
SWE-agent (open-source)
10 counted independent performance sources (8 peer-reviewed). Noted uncertainties from the entry page:
- Scope: many 2026 benchmarks use mini-SWE-agent, which is a separate product and not covered. Including it would add sources but change what is being rated.
- SWE-Bench Pro's venue was confirmed as ICML 2026 (PMLR v306); a LIPIcs record found during research belongs to a different paper.
Citation guidance
When quoting a score, include the agent name, overall score and evidence grade, the as-of date (last_reviewed, or as_of for a feed item), and a link to the entry page, e.g. "Aider (open-source): 52/100, evidence grade A, as of 2026-10-17 (Agent Reputation & Trust Clearinghouse, CC BY 4.0, https://agent-trust-clearinghouse.vercel.app/agents/aider.html)". Say that the score is informational and not a guarantee, endorsement, certification or ranking. Do not present directory or feed order as a ranking. More examples: https://agent-trust-clearinghouse.vercel.app/citing.html
More
- Methodology (factors, weights, evidence grades)
- Changelog and update feed
- For agents / data (JSON, CSV, license)