Devin (Cognition)
- Category
- Coding agent
- Type
- Proprietary cloud coding agent from Cognition AI: works in its own cloud virtual machine with shell, editor and browser, takes tasks from the web app, Slack, Jira or GitHub, and opens pull requests
- Source
- Product website
- Last reviewed
- 2026-10-17 (what this means)
Citation check incomplete — recheck later.
Summary
Devin is a proprietary cloud coding agent from Cognition AI that works in its own virtual machine and opens pull requests. Twelve independent sources that tested it were found (five peer-reviewed): eleven field studies of its public GitHub pull requests in 2024-2025 and an industry evaluation. Pull-request acceptance ranged from 30% to 68.0% and was usually among the lowest of the five agents compared, although it rose over time; an informal month-long review (not counted) reported 3 successes in 20 tasks. Its March 2024 launch demo was disputed, a 2024 access issue was fixed, and 2025 security research showed prompt-injection attacks that leaked data. No legal or regulatory action was found. Informational only; not a guarantee, endorsement, or certification.
Overall score: 50 · Evidence grade: B · Independent sources: 12 performance (5 peer-reviewed), 7 other
Evidence grade describes the amount and type of evidence, not the agent's quality.
Grade note: Grade B rather than A: 12 counted sources (5 peer-reviewed), but none states a version or is within the last 12 months, and eleven are field studies largely from one dataset (AIDev, December 2024 to July 2025), so they are not independent samples. (see methodology)
Factor breakdown
| Factor | Weight | Score | Note |
|---|---|---|---|
| Completion rate | 30% | Pull-request acceptance: 30% for performance fixes, 42.9% for fix-related work, 43.0% estimated (lowest of five), 49%, 52.12% for security work, 53.76%, 55.5% and 68.0% in the common May-July 2025 window (MSR 2026), usually below Codex, Cursor and Claude Code and close to Copilot. Acceptance rose about 0.77 points per week. An informal Answer.AI review (not counted) reported 3 of 20 tasks succeeding (January 2025). Anthar Study: highest correctness on Python tasks. | |
| Disputes | 20% | The March 2024 launch demo of an Upwork job was disputed by a software engineer and by the client; Cognition said Devin "is often inefficient and makes mistakes". CVE-2024-56083 (session live-share URL gave write access) was fixed on 2024-12-12. Prompt-injection findings were reported to Cognition in April 2025; the researcher says follow-up questions went unanswered. No legal or regulatory action was found. SWE-bench 13.86% is a self-report (not counted). | |
| Adherence to limits | 25% | Security research (2025) showed that injected instructions could make Devin run downloaded code, leak secrets and expose ports; Devin had unrestricted internet access by default at the time. Merged Devin pull requests were reverted more often than human ones (14.5% vs 11.5%). Devin changed CI/CD files most often (4.83%). Rejections often came from Devin closing its own inactive pull requests (32.1%). | |
| Evidence quality | 25% | Twelve independent sources, five peer-reviewed (four MSR 2026 papers and one Information and Software Technology article). Eleven are field studies largely from one dataset (AIDev, December 2024 to July 2025), so they are not independent samples; the twelfth is an industry report with a disclosure; none states a version and none is within 12 months. Grade B. |
0.30 × 40 + 0.20 × 45 + 0.25 × 40 + 0.25 × 75 = 49.75, rounded to 50 (rounding rule).
Scope
The Devin cloud agent from its March 2024 announcement through Devin 2.x (Devin 2.0 released April 2025). No counted study states which Devin version it used. Windsurf (acquired by Cognition in July 2025), DeepWiki and other Cognition products are not covered. Results depend heavily on the language model and task.
How sources were counted
Cognition's own reports, including the self-reported 13.86% SWE-bench result from March 2024, are not counted. No counted study has Cognition authors. Pull-request field studies count; most use the same AIDev dataset, so they are not independent samples. Commentary on the 2024 launch demo, security research and informal reviews (such as the Answer.AI month-long review) are listed as other sources.
Security and disputes
Devin runs commands and a browser in its own cloud machine. CVE-2024-56083 (CVSS 8.1) reported that a session's live-share URL could give write access to code; it was fixed on 12 December 2024. A security researcher reported prompt-injection attacks to Cognition in April 2025 and published them in August 2025, saying follow-up questions went unanswered. The March 2024 launch video of an Upwork job was disputed by a software engineer and by the client; Cognition said Devin is often inefficient and makes mistakes. No legal or regulatory action was found.
Uncertainties
- Most counted sources share the AIDev pull-request data from 2024-2025; Devin 2.x and later models may behave differently.
- No counted study states the Devin version.
- Acceptance rates depend on repository, task type and reviewer behaviour.
- The 2025 security findings were not confirmed as fixed in the sources found.
- Citation check incomplete — recheck later (Semantic Scholar was rate-limited; OpenAlex was used).
Sources
Independent performance sources (counted)
- Comparing AI Coding Agents (Trieste, KCL, UCL; MSR 2026, arXiv 2602.08915): Devin pull requests accepted at 68.0% in the common May-July 2025 window (Codex 79.9%); Devin was the only agent whose acceptance rose steadily (+0.77 points per week over 32 weeks). Field study — accessed 2026-10-17
- Where Do AI Coding Agents Fail? (Drexel, Missouri S&T; MSR 2026): Devin merge rate 53.76% (Codex highest; Copilot 43.04%). Field study — accessed 2026-10-17
- Why Are AI Agent-Involved Pull Requests (Fix-Related) Remain Unmerged? (Saskatchewan; MSR 2026, arXiv 2602.00164): fix-related Devin pull requests merged at 42.9% (Codex 81.6%, Copilot 42.4%). Field study — accessed 2026-10-17
- When AI Agents Touch CI/CD Configurations (Trent; MSR 2026, arXiv 2601.17413): Devin changed CI/CD files most often of five agents (4.83% of changes; Codex 2.01%). Field study — accessed 2026-10-17
- Security in the Age of AI Teammates (Notre Dame; Information and Software Technology 2026, arXiv 2601.00477): security-related Devin pull requests merged at 52.12% (Codex 86.59%, Copilot 49.60%). Field study — accessed 2026-10-17
- The Rise of AI Teammates in SE 3.0 (Queen's University; AIDev dataset; preprint): Devin pull requests accepted at 49% (Codex 64%, Copilot 35%). Field study — accessed 2026-10-17
- Why Agentic-PRs Get Rejected (Kyushu, Waterloo; preprint): Devin acceptance 55.5%; 32.1% of rejected Devin pull requests were closed for inactivity, consistent with Devin auto-closing inactive pull requests. Field study — accessed 2026-10-17
- How Do AI Coding Agents Contribute to Software Development? (Polytechnique Montréal; preprint Jul 2026): estimated merge probability 43.0% for Devin, the lowest of five agents (Claude Code 84.3%). Field study — accessed 2026-10-17
- Merged, Not Measured (preprint Sep 2026): Devin performance-fix pull requests accepted at 30%, the lowest of six agents; acceptance was near zero in January-April 2025, when almost all fixes in the set were Devin's. Field study — accessed 2026-10-17
- Not All Agents Are Equal (Texas Tech; preprint Sep 2026): merged Devin pull requests were reverted more often than human ones (14.5% vs 11.5%, odds ratio 1.31). Field study — accessed 2026-10-17
- Trust but Verify? (UTSA; preprint Jul 2026): 39.7% of Devin pull requests carried a security smell (corpus average 38.9%). Field study — accessed 2026-10-17
- Anthar Study (Deccan AI; industry report): 43 real pull-request tasks; Devin had the highest correctness on Python tasks (95.3% correlation with reference patches); possible context leakage between tasks was observed in IDE-based agents including Devin. Disclosure: Deccan AI sells evaluation data — accessed 2026-10-17
Other sources (not counted)
- Thoughts On A Month With Devin (Answer.AI; blog, January 2025): of 20 real tasks, 3 succeeded, 14 failed and 3 were inconclusive; outcomes were hard to predict. Informal hands-on review, listed for context (not counted) — accessed 2026-10-17
- CVE-2024-56083: before 2024-12-12, anyone who found a session's VS Code live-share URL could get write access to code (CVSS 8.1); fixed by Cognition on 2024-12-12 — accessed 2026-10-17
- Embrace The Red (Johann Rehberger, August 2025): indirect prompt injection led Devin to download and run malware, leak secrets through several channels and expose ports to the internet; reported to Cognition on 6 April 2025, and the researcher says follow-up questions about fixes went unanswered. Security research, not a performance test — accessed 2026-10-17
- March-April 2024 launch-demo dispute: a software engineer (Internet of Bugs, April 2024) reported that the "Devin's Upwork Side Hustle" video did not show the Upwork task as specified being completed; the client said only part of the task was done. Cognition said Devin "is often inefficient and makes mistakes". Reported by The Pragmatic Engineer — accessed 2026-10-17
- The original Cognition video "Devin's Upwork Side Hustle" (March 2024), cited for the dispute above — accessed 2026-10-17
- All Smoke, No Alarm (Kennesaw State and others; preprint): 26% of Devin test-file patches had strong oracles. Descriptive analysis — accessed 2026-10-17
- Habituation at the Gate (preprint Jun 2026): reviewer approval of Devin pull requests rose 3.8 points with repeated exposure (not significant). Descriptive — accessed 2026-10-17
Not counted
- Cognition launch post and technical report (March 2024): self-reported 13.86% unassisted on a 25% subset of SWE-bench; later Cognition performance reviews and customer figures. Vendor, not counted — accessed 2026-10-17
- Studies of Windsurf and other Cognition products (out of scope).
Decision record: 0045.