Skyvern (open-source)

Summary

Skyvern is an open-source browser-automation agent (AGPL-3.0) from Skyvern AI that uses language and vision models to fill forms and carry out web tasks; a hosted Skyvern Cloud service is also sold. Three independent evaluations were found, two January 2026 preprints by an overlapping Fudan University group, and a peer-reviewed prompt-injection study (ISSTA 2026) by KAIST, Seoul National University and Oregon State University. With GPT-4o, Skyvern completed 84.4% of 500 benign WebVoyager tasks. In the same study, 62.6% of social-engineering attacks succeeded on average. A separate security test scored it 61.67-72.29 (100 = no successful attacks); it was the lowest of four frameworks with GPT-4o. Skyvern's own results (85.85% on a modified WebVoyager set; Web Bench, which Skyvern co-built and ran on its own infrastructure) are not counted. Two template-injection vulnerabilities allowing code execution by authenticated or workflow inputs (CVE-2025-49619, CVE-2026-82447) were fixed in later releases. Informational only; not a guarantee, endorsement, or certification.

Overall score: 44 · Evidence grade: B- · Independent sources: 3 performance (1 peer-reviewed), 4 other

Evidence grade describes the amount and type of evidence, not the agent's quality. Grade B-: 3 counted sources (3–4 band; see methodology).

Factor breakdown

0.30 × 50 + 0.20 × 40 + 0.25 × 30 + 0.25 × 55 = 44.25, rounded to 44 (rounding rule).

Scope and version note

This entry covers the open-source Skyvern library only (AGPL-3.0; PyPI releases 0.1.x through 1.0.56, 14 October 2026), run with a language model chosen by the user. Results depend on that model; the counted tests used GPT-4o, GPT-4.1, o3 and Claude 4 Sonnet. No counted study states the Skyvern version; the tests used the GitHub code between October 2025 and January 2026. The hosted Skyvern Cloud service is not covered. The Skyvern 2.0 planner/validator design (January 2025) and the 0.1.x to 1.0.x releases are treated as one continuous product.

Shared-author disclosure

The two counted studies (“When Bots Take the Bait” and WebTrap Park) are by overlapping authors at Fudan University. They are counted as separate studies because they use different task sets and measures. The third counted study, AgentBreaker (ISSTA 2026), is by a separate group (KAIST, Seoul National University, Oregon State University).

Security

Two vulnerabilities have been published:

• CVE-2025-49619: server-side template injection (Jinja2) in workflow block prompt fields in Skyvern through 0.1.85, allowing an authenticated user to execute code on the server. CVSS 3.1 base score 8.5 as assigned by MITRE. NVD status: Deferred. Fixed by commit db856cd. A public exploit has been published.
• CVE-2026-82447: Jinja sandbox escape in TextPromptBlock before 1.0.45 (the prompt was rendered a second time without the sandbox), allowing code execution via workflow parameters or upstream block output. CVSS 3.1 8.8 and CVSS 4.0 8.7 as assigned by VulnCheck. NVD status: Deferred. Fixed in 1.0.45.

The project has not published any GitHub security advisories. In the three counted studies, social-engineering and prompt-injection attacks succeeded often (see the factor notes).

Benchmark-rewording criticism

Emergence AI, which develops the competing Agent-E, reported in its Emergence WebVoyager paper that Skyvern’s self-run WebVoyager evaluation removed 8 tasks and reworded others in ways that made them easier (for example, “five-star” changed to “at least 4.5-star”). This is Emergence’s finding; Emergence did not run Skyvern itself.

Privacy note

Skyvern Cloud is out of scope. Its privacy policy and trust centre were not reviewed, and its data handling has not been independently verified.

Uncertainties

• Three independent evaluations exist; two share authors and none states the Skyvern version. Most use GPT-4o or GPT-4.1.
• AgentBreaker ran on saved static copies of websites, not live sites, and its attacks were optimised against each agent.
• The scores rest on the minimum of three counted sources; a correction to any one would return the entry to Insufficient evidence.
• Results depend on the language model; most independent tests used GPT-4o.
• Skyvern Cloud (the hosted service) was not covered; its privacy policy and trust centre (skyvern.com/privacy, trust.skyvern.com) were not reviewed and data handling is unverified.
• Web Bench is the largest test of Skyvern but was co-built and run by Skyvern.

Sources

Independent performance sources (counted)

• "When Bots Take the Bait" (preprint, Jan 2026; Fudan University, Zhongguancun Lab, Tsinghua): with GPT-4o, Skyvern completed 84.4% of 500 benign WebVoyager tasks without defences (Browser Use 86.2%, Agent-E 84.0%). Social-engineering attack success averaged 62.6% (range 43-78% across inducement contexts; 49-76% across consistency patterns). Shares authors with WebTrap Park — independent performance, accessed 2026-10-17
• WebTrap Park (preprint, Jan 2026; Fudan University; 1,226 live-web security tasks; GitHub code accessed 11 Jan 2026): score = 1 - attack success rate. Skyvern 72.29 (Claude 4 Sonnet), 70.44 (o3), 61.67 (GPT-4o; lowest of four frameworks at that model). Sub-scores with GPT-4o: misuse 78.6, prompt injection 71.16, deceptive web design 35.25. Shares authors with Bots Take the Bait — independent performance, accessed 2026-10-17
• AgentBreaker (Son et al.; KAIST, Seoul National University, Oregon State University; Proc. ACM Softw. Eng. 3, ISSTA 2026, Article ISSTA074; peer-reviewed): indirect prompt injection against five web agents with GPT-4.1 on static copies of 60 sites sampled from Online-Mind2Web. Adaptive attacks succeeded 96.7% of the time on Skyvern, averaged over four attack goals (100% on the other four agents); task success without attack 33.3% (28.3% with the authors' defence); proposed mitigations reduced attack success to low single digits. Skyvern repository accessed 7 Oct 2025; no release stated — independent performance, accessed 2026-10-19; ACM page timed out on automated fetch; full text read from the authors' copy

Other sources (not counted)

• "Whose Agent Are You?" (preprint, June 2026; UMass Amherst): network and behavioural fingerprinting of six web agents; Skyvern (GPT-5-mini) uniquely identifiable by its TLS/HTTP2 fingerprint (17 TLS extensions). About detectability, not task performance — other, accessed 2026-10-17
• CVE-2025-49619: server-side template injection (Jinja2) in workflow block prompt fields, Skyvern through 0.1.85, authenticated users, blind remote code execution; CVSS 3.1 8.5 (MITRE); fix commit db856cd; NVD Deferred — other, accessed 2026-10-17
• CVE-2026-82447: Jinja sandbox escape in TextPromptBlock (prompt rendered twice, second time unsandboxed) before 1.0.45, via workflow parameters or upstream block output; CVSS 3.1 8.8 / 4.0 8.7 (VulnCheck); fixed in 1.0.45; NVD Deferred — other, accessed 2026-10-17
• Emergence WebVoyager (ICLR 2026 workshop; Emergence AI, which makes the competing Agent-E): does not run Skyvern but documents that Skyvern's WebVoyager run removed 8 tasks and rewrote others to be easier (e.g. "five-star" changed to "at least 4.5-star"). Context for the vendor figure; competing-author disclosure — other, accessed 2026-10-17

Excluded (vendor co-built)

• Web Bench (Halluminate and Skyvern, May 2025): Skyvern 2.0 64.4% overall (Browserbase 60.7%), 46.6% on non-read tasks; Anthropic CUA 66.0%. Co-built and co-published by Skyvern, run on Skyvern infrastructure. Excluded as vendor-affiliated — excluded (vendor co-built), accessed 2026-10-17

Vendor statements (not counted)

• Skyvern 2.0: 85.85% on a modified 635-task WebVoyager set, run in Skyvern Cloud with GPT-4o / GPT-4o-mini, self-graded; runs published at eval.skyvern.com. Not counted — vendor or maintainer, accessed 2026-10-17
• Skyvern-hosted evaluation pages ("Odysseys" model comparisons). Not counted — vendor or maintainer, accessed 2026-10-17

Checked with no Skyvern test found: Online-Mind2Web leaderboard, BrowseComp, WebArena, HAL, Mind2Web 2, VPI-Bench, AdInject, WebGuard.

Decision records: 0027; 0076 (evidence pass 2).