GitHub Copilot coding agent
- Category
- Coding agent
- Type
- Cloud coding agent from GitHub (Microsoft): assigned an issue or task, it works in a GitHub Actions environment and opens a pull request for review; proprietary; runs models chosen by GitHub or the user
- Source
- Product documentation
- Last reviewed
- 2026-10-17 (what this means)
Citation check incomplete — recheck later.
Summary
GitHub Copilot coding agent is a cloud agent from GitHub that takes an assigned issue, works in a GitHub Actions environment and opens a pull request for human review. Twelve independent field studies of its public GitHub pull requests were found (five peer-reviewed), all drawn from 2024-2025 activity. Acceptance rates ranged from 35% to 68.0% and were the lowest or near the lowest of the five agents compared in most studies; Copilot pull requests also drew the most reviews. No benchmark test of the cloud agent by independent researchers was found, and no study states a version. No CVE naming the cloud agent was found; CVEs for Copilot in IDEs and Copilot CLI are outside this entry. Informational only; not a guarantee, endorsement, or certification.
Overall score: 57 · Evidence grade: B · Independent sources: 12 performance (5 peer-reviewed), 6 other
Evidence grade describes the amount and type of evidence, not the agent's quality.
Grade note: Grade B rather than A: 12 counted sources (5 peer-reviewed), but all are field studies largely drawn from one dataset (AIDev, December 2024 to July 2025), none states a version, and none is within the last 12 months. (see methodology)
Factor breakdown
| Factor | Weight | Score | Note |
|---|---|---|---|
| Completion rate | 30% | Field studies of public pull requests: Copilot acceptance 35% (AI Teammates), 42.4% for fix-related work, 43.04%, 49.60% for security work, 50% for performance fixes, 55.0%, 59.6% estimated, and 68.0% in the common May-July 2025 window (MSR 2026). It was the lowest or near the lowest of the five agents in most comparisons (Codex usually highest). Its CI/CD changes merged more often than its other changes (+15.63 points). No independent benchmark of the cloud agent was found. | |
| Disputes | 20% | No CVE naming the cloud coding agent and no legal or regulatory action specific to it was found. About 25 CVEs name Copilot in IDEs, Copilot Chat or Copilot CLI (outside this entry). The 2022 Copilot litigation concerns code completions (outside this entry). | |
| Adherence to limits | 25% | Works only through pull requests that a human must approve. Trust but Verify?: Copilot pull requests carried a security smell most often (45.5%). Copilot pull requests drew the most reviews and change requests; Copilot made about 95% of the follow-up fixes on its merged pull requests. No direct measure of out-of-scope actions by the cloud agent was found. | |
| Evidence quality | 25% | Twelve independent sources, five peer-reviewed (four MSR 2026 papers and one Information and Software Technology article). All are field studies largely drawn from one dataset (AIDev) covering December 2024 to July 2025, so they are not independent samples, none states a version, and none is within 12 months. Grade B. |
0.30 × 45 + 0.20 × 60 + 0.25 × 50 + 0.25 × 75 = 56.75, rounded to 57 (rounding rule).
Scope
The cloud Copilot coding agent that opens pull requests (public preview May 2025; pull requests authored as copilot-swe-agent). Copilot code completions, Copilot Chat and agent mode in the IDE, Copilot CLI, Copilot code review, and the 2021 Codex-era Copilot studies are not covered. Results depend heavily on the language model and task.
How sources were counted
GitHub and Microsoft reports, and studies with GitHub or Microsoft (including Microsoft Research) authors, are not counted. Pull-request field studies of the cloud agent count. Studies of Copilot completions, Copilot in the IDE, Copilot CLI or Copilot code review are not evidence about this product.
Security
The coding agent runs in a GitHub Actions environment and its changes reach a repository only through a pull request that a person approves. No CVE naming the cloud coding agent was found. About 25 CVEs published in 2025 and 2026 name GitHub Copilot in Visual Studio Code or Visual Studio, Copilot Chat or Copilot CLI; those products are outside this entry. No legal or regulatory action specific to the coding agent was found.
Uncertainties
- All counted sources are field studies, mostly of the same AIDev pull-request data from 2024-2025; the product and its default models have changed since.
- Acceptance rates depend on repository, task type and reviewer behaviour, not only on the agent.
- No version is stated in any study, and none tested the agent after July 2025.
- Copilot in IDEs and Copilot CLI have their own evidence and CVEs; they may need separate entries.
- Citation check incomplete — recheck later (Semantic Scholar was rate-limited; OpenAlex was used).
Sources
Independent performance sources (counted)
- Comparing AI Coding Agents (Trieste, KCL, UCL; MSR 2026, arXiv 2602.08915): GitHub Copilot pull requests accepted at 68.0% in the common May-July 2025 window (Codex 79.9%, Cursor 74.4%, Claude Code 72.6%, Devin 68.0%); Copilot pull requests received the most reviews (4.94 per pull request). Field study — accessed 2026-10-17
- Where Do AI Coding Agents Fail? (Drexel, Missouri S&T; MSR 2026): Copilot had the lowest merge rate of five agents, 43.04% (Codex highest). Field study — accessed 2026-10-17
- Why Are AI Agent-Involved Pull Requests (Fix-Related) Remain Unmerged? (Saskatchewan; MSR 2026, arXiv 2602.00164): fix-related Copilot pull requests merged at 42.4% (Codex 81.6%, Devin 42.9%). Field study — accessed 2026-10-17
- When AI Agents Touch CI/CD Configurations (Trent; MSR 2026, arXiv 2601.17413): agent pull requests changing CI/CD merged less often overall (67.77% vs 71.80%), except Copilot, whose CI/CD changes merged 15.63 points more often. Field study — accessed 2026-10-17
- Security in the Age of AI Teammates (Notre Dame; Information and Software Technology 2026, arXiv 2601.00477): security-related Copilot pull requests merged at 49.60%, the lowest of five agents (Codex 86.59%). Field study — accessed 2026-10-17
- The Rise of AI Teammates in SE 3.0 (Queen's University; AIDev dataset; preprint): Copilot pull requests accepted at 35% (Codex 64%, Devin 49%); 42.2% of Copilot pull requests were bug fixes. Field study — accessed 2026-10-17
- Why Agentic-PRs Get Rejected (Kyushu, Waterloo; preprint): Copilot acceptance 55.0% (Codex 85.8%, Devin 55.5%, human 82.6%). Field study — accessed 2026-10-17
- How Do AI Coding Agents Contribute to Software Development? (Polytechnique Montréal; preprint Jul 2026): estimated merge probability 59.6% for Copilot (Claude Code 84.3%, Codex 73.5%, Devin 43.0%). Field study — accessed 2026-10-17
- Merged, Not Measured (preprint Sep 2026): performance-fix pull requests accepted at 50% for Copilot (Codex, Claude Code, Jules 73%; Devin 30%); Copilot acceptance fell from 61% to 42% as repository size grew. Field study — accessed 2026-10-17
- Who Finishes the Job? (Empirical Software Engineering submission; preprint Sep 2026): Copilot made about 95% of the follow-up fixes on its own merged pull requests; humans most often fixed Codex merges (38.4%). Field study — accessed 2026-10-17
- Not All Agents Are Equal (Texas Tech; preprint Sep 2026): post-merge quality across five agents; Copilot pull requests drew the most human reviews and change requests. Field study — accessed 2026-10-17
- Trust but Verify? (UTSA; preprint Jul 2026): Copilot pull requests carried a security smell most often, 45.5% (corpus average 38.9%; Codex 34.9%). Field study — accessed 2026-10-17
Other sources (not counted)
- All Smoke, No Alarm (Kennesaw State and others; preprint): 26% of Copilot test-file patches had strong oracles (Claude Code 41%). Descriptive analysis — accessed 2026-10-17
- Habituation at the Gate (preprint Jun 2026): reviewer approval rates for Copilot pull requests rose by 9.1 points with repeated exposure, the largest rise of four agents. Descriptive analysis of review behaviour — accessed 2026-10-17
- What Makes a GitHub Issue Ready for Copilot? (ÉTS Montréal; preprint): issue-quality model predicting merged Copilot pull requests (median AUC 72%). Descriptive — accessed 2026-10-17
- Why Are Agentic Pull Requests Merged or Rejected? (Waseda and others; preprint): Copilot and Devin pull requests were more often reviewer-mediated. Descriptive — accessed 2026-10-17
- AgentLogs (Radboud, NAIST; preprint Aug 2026): dataset of Copilot coding agent session logs. Dataset, no outcome measure — accessed 2026-10-17
- About 25 CVEs (2025-2026) name GitHub Copilot in Visual Studio Code or Visual Studio, Copilot Chat, or Copilot CLI (for example CVE-2025-53773, CVE-2026-29783, CVE-2026-45033). None found names the cloud coding agent; these products are outside this entry's scope — accessed 2026-10-17
Not counted
- GitHub and Microsoft documentation, launch posts and reports for Copilot coding agent; vendor, not counted — accessed 2026-10-17
- Investigating Autonomous Agent Contributions in the Wild (TU Delft, UC Davis, GitHub): has a GitHub co-author, so not counted for Copilot — accessed 2026-10-17
- Agentic Coding in the Wild: Characterizing GitHub Copilot at Production Scale (UIUC, Microsoft Azure Research): Microsoft co-authors; not counted — accessed 2026-10-17
- Studies of Copilot code completions, including 2021-2023 Codex-era studies, Copilot in the IDE, Copilot CLI and Copilot code review (out of scope).
Decision record: 0044.