Agent S (open-source framework)

Summary

Agent S is an open-source framework from Simular that lets a language model operate a desktop computer by looking at the screen and using the mouse and keyboard. Results depend heavily on the model and settings used. In independent evaluations of S2, S2.5 and S3 (OS-Symphony, ACL 2026; OSWorld-Human, MLSys 2026; MacAgentBench; Agent Alpha; OS-Marathon; SCUBA; WorldGUI), success ranged from about 10% to 67% depending on the benchmark and model: 40-55% on reproduced OSWorld runs, 34-40% on Windows and Salesforce tasks, and 23-34% on long repetitive tasks (humans 80-88%). These are below the vendor's own OSWorld figures of 62.6-72.6%. Three low-to-medium severity denial-of-service CVEs were published in September 2026 with no vendor response recorded. An independent code review found that the command-line app runs model-generated actions without asking for confirmation. An independent preprint found that web pages changing at the moment of action could redirect it in all 14 cases tested, as for other screenshot-based agents. Simular's commercial products are not covered. Overall score 49. Informational only; not a guarantee, endorsement, or certification.

Overall score: 49 · Evidence grade: B · Independent sources: 8 performance (2 peer-reviewed), 5 other

Evidence grade describes the amount and type of evidence, not the agent's quality. Grade B rather than A: 7 counted sources but only 2 peer-reviewed (see methodology).

Factor breakdown

0.30 × 45 + 0.20 × 45 + 0.25 × 35 + 0.25 × 70 = 48.75, rounded half up to 49.

Scope

This entry covers only the open-source Agent S framework (PyPI gui-agents 0.2.x–0.3.2: Agent S2, S2.5 and S3) run with a language model and grounding model chosen by the user. Simular’s commercial products (the Simular desktop app, Sai, Simular Cloud and the Simular API) are not covered.

Version note

Agent S2 (March 2025), S2.5 (August 2025) and S3 (October 2025; current release 0.3.2, December 2025) are treated as one continuing framework: same repository, package and team, with each generation built on the previous one. All generations still ship in the same package. Results are labelled by generation below. Agent S1 (October 2024) is historical context only. S3 removed S2’s hierarchical planning and added a coding agent and an optional “Behavior Best-of-N” mode, which runs several attempts and picks the best.

Independent results by model

Results depend heavily on the model, step limit and number of attempts. Simular’s own OSWorld figures (62.6–72.6%) are not counted.

Security and disputes

• Three denial-of-service vulnerabilities in 0.3.1/0.3.2 were published on 3 September 2026 by the VulDB CNA: CVE-2026-84885 (CVSS 3.1 4.3), CVE-2026-84886 (5.3) and CVE-2026-84887 (4.3). According to VulDB, the vendor did not respond. NVD status: Deferred. No fixed release was found.
• Reproduction gap: in GitHub issue #146 (October 2025), a user got lower S3 results than the paper. Simular traced this to a prompt change missing from the open-source code and fixed it on 30 October 2025; the user confirmed the fix worked.
• Confirmation before actions: an independent code review reports that the command-line app runs model-generated actions without asking for confirmation. In the source at commit 15776a8, a permission dialog is defined but not called. Optional local code execution is off by default and comes with a warning.
• One independent safety test (TOCTOU study, preprint): page changes timed between the agent's check and its action caused unintended actions in all 14 cases tested. No independent prompt-injection test of Agent S was found. No legal or regulatory action was found.

Uncertainties

• Results depend on the model, grounding model, step limit and number of rollouts; no single figure describes "Agent S".
• S2, S2.5 and S3 differ materially in design. Treating them as one product follows the library precedent and is a judgment call (decision 0025).
• OSWorld-Verified rows may be based on trajectories supplied by Simular; they are not counted.
• The only independent safety test (TOCTOU, preprint) does not state the Agent S version or model, and its result is shared by all screenshot-based agents tested; no independent prompt-injection evaluation was found.
• Simular's commercial products (Simular app, Sai, Cloud, API) are not covered.

Sources

Independent performance sources (counted)

• OS-Symphony (ACL 2026; Shanghai AI Lab and others; arXiv 2601.07779): authors reproduced Agent S3 on OSWorld-Verified (361 tasks, 50 steps): GPT-5-Mini 47.58%, Qwen3-VL-32B-Instruct 40.11%. Their 62.63% figure for S3 with GPT-5 is taken from Simular (not counted). Competing-method disclosure: the authors propose a rival framework — accessed 2026-10-17
• OSWorld-Human (MLSys 2026; UC San Diego): ran Agent S2 on 39 OSWorld tasks and found planning and reflection calls were 76-96% of task latency. Efficiency-weighted scores: S2 with Gemini 2.5 15.6% single-action / 9.6% grouped (original 41.4%); S2 with Claude 3.7 6.6% / 4.4% (original 34.5%). These use published leaderboard trajectories; S3 was not studied — accessed 2026-10-17
• MacAgentBench (preprint, June 2026; Shanghai Jiao Tong / Shanghai AI Lab): Agent S3 on 676 macOS tasks, Pass@1 9.6% (InternVL3.5-8B) to 66.9% (Claude Opus 4.6); 1.8-50.3% when all 4 attempts must succeed; GPT-5.4 58.9%. Overlap disclosure: shares several authors with OS-Symphony; OpenClaw scored 73.7% in the same study — accessed 2026-10-17
• Agent Alpha (preprint, Feb 2026; George Washington University): authors ran Agent S3 with GPT-5-mini (Best-of-3, 30 steps) on OSWorld: 54.29% success, 313 s average per task. Competing-method disclosure: Agent Alpha scored 64.27%. The 72.58% S3 figure in their Table 1 is copied from the leaderboard (not counted) — accessed 2026-10-17
• OS-Marathon (preprint v3, Aug 2026; Oxford, Microsoft and others): Agent S2.5 with GPT-5 scored 33.50% field-level / 23.44% subtask-level on long repetitive tasks (humans 88.38/80.52; GPT-5.4 native computer use 68.29/51.39). Disclosure: Microsoft co-authors; Simular is a Windows 365 for Agents pilot partner — accessed 2026-10-17
• SCUBA (preprint, Sept 2025; Salesforce AI Research): Agent S2.5 with GPT-5 on 300 Salesforce CRM tasks, 39.00% success zero-shot and 40.38% with demonstrations; about 25 min and $1.17 per task (grounder cost excluded). Disclosure: Salesforce sells its own agent products — accessed 2026-10-17
• WorldGUI (preprint v5, May 2026, under submission; NUS Show Lab): Agent S2 34.2% overall success on Windows desktop tasks started from varied states (humans 85.3%). Competing-method disclosure: their WorldGUI-Agent scored 45.8% with GPT-5.1. Its WindowsAgentArena table reuses vendor numbers (not counted) — accessed 2026-10-17
• TOCTOU study (Ohio State University; preprint, Feb 2026): tested 10 open-source browser- and computer-use agents on 14 cases where a web page changes between the agent's check and its action (layout changes, data updates, expiring states; 9 synthesized pages, 5 real sites). The vulnerability was triggered in all 14 cases for Agent S, as for the six other screenshot-based agents; agents with structured page access resisted layout changes. Version and model not stated. Funding: NSF and Schmidt Sciences; the authors propose a mitigation. — accessed 2026-10-17

Other sources (not counted)

• OSWorld-Verified leaderboard (XLANG Lab, HKU): Agent S3 rows 72.58% (Opus 4.5 + GPT-5, Best-of-10), 69.90% (GPT-5, Best-of-10), 67.46% and 65.58% (single); Agent S2.5 with o3 56.0%. Maintainers either rerun submitted code or accept trajectories from "trusted institutions"; which applied to Simular is not stated. Simular helped fix OSWorld-Verified tasks, and a Simular author is on OSWorld 2.0 — accessed 2026-10-17
• Public reproduction attempt (Oct-Nov 2025): a user got lower S3 results (for example about 208/361 with Azure GPT-5). On 30 Oct 2025 Simular said a prompt commit had been missing from the open-source code (aaed16e), and the user confirmed the fix worked — accessed 2026-10-17
• CVE-2026-84885 / -84886 / -84887 (published 3 Sept 2026, VulDB CNA): denial of service in code_agent.py, the s1 OCR server and grounding.py of 0.3.1/0.3.2; CVSS 3.1 4.3 / 5.3 / 4.3; vendor did not respond per VulDB; NVD status Deferred — accessed 2026-10-17
• Independent code review of gui-agents 0.3.2 (greenlitbooks field note): the agent acts without asking for confirmation, the permission dialog is unused, there is no sandbox, and coding mode is off by default — accessed 2026-10-17
• Source code at commit 15776a8 (8 Oct 2026): the command-line app executes model-generated actions directly; a permission dialog function is defined but not called — accessed 2026-10-17

Not counted

• Agent S3 paper (Simular; TMLR 2026): OSWorld 62.6% (GPT-5), 69.9% (Best-of-10), 72.6% (GPT-5 + Opus 4.5); WindowsAgentArena 56.6%; AndroidWorld 71.6%. Not counted — accessed 2026-10-17
• Agent S2 paper (Simular; COLM 2025): author-run OSWorld/WindowsAgentArena/AndroidWorld results. Not counted — accessed 2026-10-17
• Agent S (S1) paper (ICLR 2025): 20.6% OSWorld, author-run. Historical context only — accessed 2026-10-17
• Sai (Simular commercial agent) 73% on OSWorld 2.0, as claimed by Simular. Sai is outside scope; not counted — accessed 2026-10-17
• Vendor figures reused inside counted papers (OS-Symphony’s S3 + GPT-5 row, Agent Alpha’s leaderboard table, WorldGUI’s WindowsAgentArena table).
• Checked with no Agent S tested: Princeton HAL, OSUniverse, OS-Harm, VPI-Bench, OSGuard, OSWorld 2.0.

Decision records: 0025; 0050 (TOCTOU safety test).