Agent Reputation & TrustClearinghouse

Claude Code (Anthropic)

Category
Coding agent
Type
Commercial agentic coding tool from Anthropic (terminal CLI, npm @anthropic-ai/claude-code, with IDE extensions); used with a paid Claude plan or API key; source not open
Source
Product documentation
Last reviewed
2026-10-17 (what this means)

Citation check incomplete — about 13 candidate studies not yet screened; recheck later.

Summary

Claude Code is Anthropic's agentic coding tool. It works in a terminal to read a codebase, edit files, run commands and open pull requests. In independent tests from 2025-26, results varied widely by task and model: 32.8% average resolved rate on SWE-RPG repository tasks, 47.9% pass@1 on SWE-bench Science (with Claude Opus 5), 58.0% on TerminalWorld terminal tasks, 61.2% on SkillsBench, and 78.6% bug recall on PBT-Bench. Two field studies of real GitHub pull requests found 72.6% and 83.8% were accepted (human pull requests 91.0% in one study). Safety results were mixed: one study found out-of-scope actions on benign tasks in up to 27.7% of runs, while another found the lowest boundary-violation rates of four frameworks on two of three task types. An independent stress test of the auto-approval mode (April 2026) found it missed 81.0% of risky actions in deliberately ambiguous tasks (Anthropic reports 17% on production traffic), and only 5.9-8.6% of solutions on a security benchmark were both working and secure. A red-teaming study achieved remote code execution on an older version (2.0.37), but not on version 2.1.138 with newer models. Anthropic has published more than 30 security advisories for Claude Code since June 2025, each with a fixed version. In March 2026, its source code was accidentally published in an npm package. Anthropic's own benchmark figures are not counted. Overall score 55. Informational only; not a guarantee, endorsement, or certification.

Overall score: 55 · Evidence grade: A · Independent sources: 17 performance (6 peer-reviewed), 7 other

Evidence grade describes the amount and type of evidence, not the agent's quality.

Factor breakdown

Factor breakdown
FactorWeightScoreNote
Completion rate30%50SWE-RPG (163 repository tasks, 6 backends): Claude Code averaged 32.8% resolved, 49.1% best (Kimi-K3). SWE-bench Science: pass@1 47.90% (Opus 5, max) and 42.02% (DeepSeek-V4-Pro). TerminalWorld: 58.0% pass (Opus 4.7), below Terminus-2 at 62.5% with the same model. SkillsBench: 61.2% with Skills (Opus 4.7), vs 53.1% in OpenHands. PBT-Bench: 78.6% bug recall (Opus 4.8), vs 81.2% in OpenHands. ResearchClawBench: 21.5 of 100 (Opus 4.6), the top of 7 research agents. SusVibes: 12.9-41.4% functional, below SWE-agent and OpenHands with Claude 4 Sonnet. Field studies: 83.8% of 567 pull requests accepted (humans 91.0%); 72.6% in a May-July 2025 window. Judgment call.
Disputes20%45No lawsuits or regulatory actions found. Anthropic's GitHub repository lists 35 security advisories (June 2025 to October 2026; 28 rated high), mostly approval-prompt or sandbox bypasses and trust-dialog issues; each lists a fixed version, and a few concern Claude Desktop rather than the CLI. Three further CVEs filed by VulnCheck in April 2026 (CVE-2026-35020/35021/35022) were later rejected by that CNA as outside the threat model or intended behaviour. On 31 March 2026, version 2.1.88 was published to npm with a source map exposing the source code; Anthropic called it a packaging error with no customer data exposed, and retracted most of the DMCA notices that had reached about 8,100 GitHub repositories. Judgment call.
Adherence to limits25%40Overeager Coding Agents (Claude Code 2.1.117): out-of-scope actions on benign tasks 5.4-27.7% depending on model, vs 1.1% for the same model in OpenHands. SWE-MISBEHAVIOR (Sonnet 4.5): boundary-violation rate 3.80% on code generation and 8.50% on file management (lowest of four frameworks), 18.95% on system interaction. Red-teaming (ISSTA 2026): remote code execution on 2.0.37; 0.0 success on 2.1.138 with Sonnet 4.6 and Opus 4.7. Measuring the Permission Gate (HKUST, ETH; April 2026): auto mode missed 81.0% of risky state-changing actions in 128 deliberately ambiguous tasks (70.3% of those it evaluated; 31.9% false positives), vs 17% reported by Anthropic on production traffic. SusVibes: 5.9-8.6% of Claude Code solutions were both functional and secure (SWE-agent up to 12.9%). Several benchmark studies ran with permission prompts disabled. Judgment call.
Evidence quality25%85Seventeen independent sources, six peer-reviewed (ISSTA 2026, TOSEM, MSR 2026, NeurIPS 2026 Datasets & Benchmarks, ASE 2026 extended abstract, ICML 2026). Recent tests state versions (2.1.117, 2.1.138, 2.1.139; 2026). Terminal-Bench 2.0 is excluded (an Anthropic co-author). Many runs use non-default models or settings. Judgment call.

0.30 × 50 + 0.20 × 45 + 0.25 × 40 + 0.25 × 85 = 55.25, rounded to 55 (rounding rule).

Scope

The Claude Code agent (CLI and headless mode), versions 1.0 to 2.1.x (latest 2.1.302, 16 Oct 2026). Claude models used in other scaffolds, the Claude Agent SDK, Claude Desktop/Cowork and Claude Research (a separate entry) are not covered. Several studies ran Claude Code with non-Anthropic models. Anthropic's research mode in the Claude apps is a separate entry: see Claude Research.

How sources were counted

Only studies that ran Claude Code itself are counted; studies of Claude models in other tools are not. Anthropic's own results are not counted. Terminal-Bench 2.0 ran Claude Code but has an Anthropic-affiliated author, so it is excluded. Two field studies of real GitHub pull requests are counted.

Disclosures for counted sources

SkillsBench thanks Anthropic, among other model providers, for model access and API support. TerminalWorld used Claude Code to help build its tasks. UrbanDS and SWE-MISBEHAVIOR propose competing systems or mitigations. Several studies ran Claude Code with permission prompts disabled or with non-Anthropic models.

Security

Anthropic has published 35 security advisories for Claude Code and related apps since June 2025, mostly approval-prompt, sandbox or trust-dialog bypasses, each with a fixed version. Three CVEs filed by VulnCheck in April 2026 were later rejected by that CNA. On 31 March 2026, version 2.1.88 was published to npm with a source map exposing the source code; Anthropic described it as a packaging error with no customer data exposed. No lawsuits or regulatory actions were found.

Uncertainties

Sources

Independent performance sources (counted)

Other sources (not counted)

Excluded and vendor sources (not counted)

Decision records: 0040; 0049 (SusVibes); 0052 (citation backfill).