Changelog
Approved changes to entries and rules, newest first. Each item links to its decision packet. Informational only.
| Date | Decision | Change |
|---|---|---|
| 2026-10-20 | 0086 | The directory moved to a new address, www.agentclearinghouse.org. Pages, data files and feeds keep the same paths, and links to the old address (agent-trust-clearinghouse.vercel.app) continue to work. Feed item ids are unchanged. No score or grade changes. |
| 2026-10-20 | 0085 | Added mini-SWE-agent (open-source) (minimal bash-only coding agent from the SWE-bench and SWE-agent team; Coding agent): overall 61, evidence grade A, 16 independent performance sources (9 peer-reviewed). Most counted studies run it alongside other harnesses with the same models; recent tests state the version tested, including the latest release (2.4.6). Studies and leaderboards involving mini-SWE-agent developers are not counted, and no runs are counted for both this entry and SWE-agent. One medium-severity CVE affects its optional Bubblewrap sandbox. Citation check incomplete — recheck later. The SWE-agent scope now notes that mini-SWE-agent has its own entry, and the coding-agents comparison lists it among open-source frameworks (no score or grade change for SWE-agent). |
| 2026-10-19 | 0084 | Added FacTool (open-source) (GAIR-NLP research code for claim-level factuality checking; Verification): overall 55, evidence grade B, 17 independent performance sources (13 peer-reviewed). Nearly all counted sources are papers proposing alternative methods that run FacTool as a comparison system, and five come from one connected group of authors that shares result rows. No study states the FacTool version tested. One study with a FacTool author is not counted. Citation check incomplete — recheck later. |
| 2026-10-19 | 0083 | Added Cursor agent (Anysphere; agent mode in the Cursor editor, background and cloud agents, and the Cursor CLI; Coding agent): overall 57, evidence grade A, 19 independent performance sources (10 peer-reviewed). Fourteen counted sources are pull-request field studies drawn largely from one dataset (AIDev, 2025), so they are not independent samples. Anysphere material and two comparisons by authors from a competing product's company are not counted. Tab completion, inline edits, chat-only use and Bugbot code review are out of scope. Citation check incomplete — recheck later. |
| 2026-10-19 | 0077 | Evidence pass on three Insufficient-evidence entries; no new counted source found, so no scores or grades change. AutoKaggle: a re-implementation of its workflow (DS-Lighting) and a paper that copied its figures (LightAutoDS-Tab) added as uncounted sources; citation recheck completed. Loki: two papers that used Loki as a tool without measuring its accuracy added as uncounted sources. UI-TARS Desktop: a model-only prompt-injection test (VPI-Bench) added as context, not counted. All three stay Insufficient evidence. |
| 2026-10-19 | 0076 | Evidence pass on three Insufficient-evidence entries. Skyvern: AgentBreaker (ISSTA 2026, peer-reviewed prompt-injection study) added as a counted source; 3 counted (1 peer-reviewed), so the entry is scored for the first time: 44, grade B-. Julius AI: a 2026 Quality & Quantity study (peer-reviewed; the vendor provided complimentary access) and a 2026 IJRSI education study (limited) added as counted sources; 4 counted (2 peer-reviewed), so the entry is scored for the first time: 59, grade B-; recheck note removed. Kimi Researcher: no new independent test found; a re-scoring of its DeepResearch Bench reports added as an uncounted source; still Insufficient evidence. |
| 2026-10-19 | 0075 | Shortened eleven page titles (browser tab and search result titles) to 70 characters or fewer, keeping the site name at the end. Headings, descriptions, data, scores and grades are unchanged. |
| 2026-10-18 | 0074 | Added llms-full.txt, all entries in one plain-text file generated from the same data as agents.json, linked from llms.txt, the data page and the How to cite page. Added a “How to cite” link to the site footer. No score or grade changes. |
| 2026-10-18 | 0072 | Added a short “How this directory works” note on the home page and a matching section on the methodology page (counted independent evidence, separate factor scores, open data, no ranking or endorsement), with links from the data page and llms.txt. Home-page last-reviewed dates are now generated from the data (four rows corrected). No score or grade changes. |
| 2026-10-18 | 0071 | Evidence pass on three Insufficient-evidence entries: Manus, ChatGPT Work and Open Interpreter. New sources and notes added; ChatGPT Work now has two counted independent tests, fewer than the three required. All three remain Insufficient evidence; no scores or grades. |
| 2026-10-18 | 0070 | Code license: the code (scripts, stylesheet and vercel.json) is now licensed under the Apache License 2.0; data and site text stay CC BY 4.0; the project name, logo and favicon are not licensed. Added a CITATION.cff file to the repository. License wording updated on For agents / data, How to cite and llms.txt. No score or grade changes. |
| 2026-10-18 | 0069 | Small consistency fixes: the same footer on every page (methodology now links For agents / data); a one-sentence scope for seven entries, taken from each entry page; a published grade note under the evidence grade for Aider, SWE-agent, Devin, GitHub Copilot coding agent and STORM; the Atom content type for updates.xml; shorter titles for two comparison pages; a shorter description for the citing page. No score or grade changes. |
| 2026-10-18 | 0067 | New How to cite and quote this data page: ready-to-copy citations (plain sentence, APA-style, BibTeX, agent quote template), CC BY 4.0 attribution, how to quote scores and machine-readable locations, with example values generated from the data. The typed-in citation example on the data page, llms.txt and comparison pages is now generated. No score or grade changes. |
| 2026-10-18 | 0066 | Three comparison pages generated from the directory data: Claude Code and OpenAI Codex compared, Coding agents compared and Research agents compared (public evidence; alphabetical; scope notes; informational, not a ranking). Category labels capitalized consistently in the data export (display only). No score or grade changes. |
| 2026-10-18 | 0065 | New agent update feed: updates.json (JSON Feed 1.1) and updates.xml (Atom) list new entries, score and grade changes and other changelog items, each with before and after values where recorded and the informational disclaimer. This changelog and both feeds are now generated from one source file. Citation guidance added to llms.txt and the data page. No score or grade changes. |
| 2026-10-18 | 0061 | New “For agents / data” page with JSON, CSV and JSON Schema exports generated from the entries; data and site text licensed under CC BY 4.0; llms.txt, robots.txt and sitemap updated; directory list marked as sorted by overall score, not a ranking of quality. Footer link added. No score or grade changes. |
| 2026-10-18 | 0060 | Methodology: new “Privacy and measurement” section (Google Search Console and hosting request statistics; aggregate monthly totals only; never used in scores). Footer link “Privacy” added to all pages. No score or grade changes. |
| 2026-10-17 | 0059 | Browser Use: AgentTell (privacy, Browser Use 0.13.1, Aug 2026), Privacy Practices of Browser Agents (Brave co-authors disclosed) and BreakingWeb (Amazon co-authors disclosed) added as counted sources; 11 counted (3 peer-reviewed). Grade B → A because a recent test now states the library version. Overall score unchanged at 51; recheck note removed. ceLLMate reviewed, not counted. |
| 2026-10-17 | 0058 | Google DeepMind SAFE: VeriFact (EMNLP 2025) added as a counted source; 4 counted (4 peer-reviewed); 56.7% of SAFE-extracted facts were incomplete on a difficult test set. Score 66 and grade B- unchanged. Consensus: no new tests found (the only lead was the already-excluded retracted paper); recheck note removed; 62, B unchanged. Julius AI: two 2025 items reviewed, none counted; Insufficient evidence unchanged. |
| 2026-10-17 | 0057 | Gemini Deep Research: MisKnow-Agent (robustness preprint, July 2026) added as a counted source; 8 counted (2 peer-reviewed); reports adopted planted false conclusions in up to 54% of cases. Scores and grade unchanged (63, B). OpenAI Deep Research: two items reviewed, none counted (one perspective, one paywalled); 65, B unchanged. Recheck notes updated. |
| 2026-10-17 | 0056 | Perplexity: Follow-up review of five 2025–2026 studies; none in scope (health-information quality, or no citation check), so none counted. Score 54 and grade B unchanged. Recheck note updated (0056). |
| 2026-10-17 | 0055 | Browser Use: Atomicity for Agents (TOCTOU security study, Ohio State, preprint 2026) added as a counted source; 8 counted (3 peer-reviewed). Vulnerability triggered in 9 of 14 test cases. Scores and grade unchanged (51, B). |
| 2026-10-17 | 0054 | Elicit: three peer-reviewed tests added (Mazzali et al. 2026, Meliante et al. 2025, Spillias et al. 2025); 10 counted sources (9 peer-reviewed). Search remains weak; screening was strong in a 2025 Pro-tier test; extraction ranged from 51% to 92% accurate. Scores unchanged; overall 68, grade A. |
| 2026-10-17 | 0053 | SWE-agent: Sifting the Noise (ISSTA 2026) added, testing a December 2025 main-branch commit; 10 counted sources (8 peer-reviewed). Grade B → A because a recent test now states the version. Overall score unchanged at 58; evidence quality stays 80. |
| 2026-10-17 | 0052 | Citation backfill: OpenHands and Claude Code sources added (Sifting the Noise; WideSWE and Timeline-Bench); OpenHands 12 counted (7 peer-reviewed), Claude Code 17 counted (6 peer-reviewed); recheck notes added to 13 entries. No score or grade changes. |
| 2026-10-17 | 0051 | Grade-A basis check: OpenAI Codex, OpenHands and Claude Code confirmed A. Elicit kept A; its basis (a dated Pro-tier test, Dec 2025 to April/May 2026) is now stated on the page. Aider kept A; Sifting the Noise recorded as peer-reviewed (ISSTA 2026), 6 peer-reviewed of 8, and its inferred test date is noted. No score changes. |
| 2026-10-17 | 0050 | Agent S: correction. The entry said no independent safety test had been found; an independent TOCTOU study (Ohio State, preprint) that tested Agent S is now counted. Now 8 independent performance sources (2 peer-reviewed). Overall score 49 and evidence grade B unchanged. |
| 2026-10-17 | 0049 | OpenHands: added SusVibes (ICML 2026), an independent security benchmark, as a counted source, with disclosures including a shared-institution (Carnegie Mellon) note; now 11 independent performance sources (6 peer-reviewed); overall 61 and grade A unchanged. Claude Code: SusVibes now recorded as peer-reviewed (ICML 2026); 6 peer-reviewed of 15; overall 55 and grade A unchanged. |
| 2026-10-17 | 0048 | SWE-agent: added SusVibes (ICML 2026), an independent security benchmark, as a counted source, with disclosures. Now 9 independent performance sources (7 peer-reviewed). Overall score 58 and evidence grade B unchanged. |
| 2026-10-17 | 0046 | Design refresh (layout and labels only; no score or rule changes) |
| 2026-10-17 | 0045 | Added Devin (Cognition; cloud coding agent, Devin 1.x-2.x; Coding agent): overall 50, evidence grade B, 12 independent performance sources (5 peer-reviewed), including an industry report with disclosure. Cognition's self-reported SWE-bench result is not counted; an informal hands-on review is listed for context only. Windsurf and other Cognition products are out of scope. Citation check incomplete — recheck later. This completes Batch 4. |
| 2026-10-17 | 0044 | Added GitHub Copilot coding agent (cloud agent that opens pull requests; Coding agent): overall 57, evidence grade B, 12 independent pull-request field studies (5 peer-reviewed). Copilot completions, IDE agent mode, Copilot CLI and code review are out of scope; GitHub- and Microsoft-authored studies are not counted. Citation check incomplete — recheck later. |
| 2026-10-17 | 0043 | Added OpenAI Codex (CLI and cloud agent; Coding agent): overall 62, evidence grade A, 26 independent performance sources (5 peer-reviewed; funding disclosures). The 2021 Codex model and GPT-Codex models run in other harnesses are out of scope; OpenAI-authored material is not counted. Citation check incomplete — recheck later. |
| 2026-10-17 | 0042 | Added AIDE (Weco AI, open-source; Data science agent): overall 60, evidence grade B, 13 independent performance sources (6 peer-reviewed; competing-method disclosures). Grade B because no test from the last 12 months states the version. Added AutoKaggle (open-source; Data science agent) as a separate entry: Insufficient evidence (2 counted). Both carry "Citation check incomplete — recheck later". |
| 2026-10-17 | 0041 | Added Aider (open-source; Coding agent): overall 52, evidence grade A, 8 independent performance sources (5 peer-reviewed). Evidence quality 80, below the suggested A range, because the A rests on a single recent versioned preprint test. Citation check incomplete — recheck later. Maintainer-run Aider leaderboards not counted. |
| 2026-10-17 | 0040 | Added Claude Code (Anthropic; Coding agent): overall 55, evidence grade A, 15 independent performance sources (5 peer-reviewed). Terminal-Bench 2.0 excluded (Anthropic-affiliated author); Anthropic results not counted; SkillsBench API-support disclosure. Citation check incomplete (about 13 candidate studies not yet screened). Claude Research now links to the Claude Code entry. |
| 2026-10-17 | 0039 | Added UI-TARS Desktop (ByteDance, open-source app; Computer-use agent): Insufficient evidence, 1 counted independent source (TOCTOU security study, preprint). Scope: app and Agent TARS; UI-TARS models not covered. Vendor results and model-only tests not counted. CVE-2026-81735 noted. |
| 2026-10-17 | 0038 | Added SWE-agent (open-source; Coding agent): overall 58, evidence grade B, 8 independent performance sources (6 peer-reviewed; competitor disclosures). Grade B because no test from the last 12 months states the version. mini-SWE-agent out of scope; developer results, developer-affiliated studies and leaderboard entries not counted. |
| 2026-10-17 | 0037 | Added OpenHands (open-source; Coding agent): overall 61, evidence grade A, 10 independent performance sources (5 peer-reviewed; competing-author and competitor disclosures). Developer results, developer-affiliated studies and leaderboard entries not counted. |
| 2026-10-17 | 0036 | Methodology: added the Coding agent category, with a definition and boundaries against Computer-use, Data science and Simple analysis. No existing entry changes category; scoring rules are unchanged. |
| 2026-10-17 | 0035 | Browser Use (open-source library): evidence grade changed from A to B for consistency with the 0025 clarification as applied to STORM. None of its recent independent tests states the library version, so a current-version test within 12 months is not shown. Overall score (51), sources and directory order are unchanged. |
| 2026-10-17 | 0031 | Added Fellou (agentic browser; Browser/web agent): Insufficient evidence, 0 counted independent performance sources. CAP excluded for Fellou (vendor-affiliated author); vendor figures listed, not counted; security reports from Brave and LayerX noted with their commercial interests. |
| 2026-10-17 | 0030 | Added Loki (open-source) and OpenFactCheck (open-source), both Verification, both Insufficient evidence. Loki: 2 independent performance sources (TAIGR; Proof-of-Guardrail), found in a Semantic Scholar citation check. OpenFactCheck: 0. Developer papers listed, not counted; shared authors noted on both pages. |
| 2026-10-17 | 0029 | Added Data Interpreter (MetaGPT, open-source; new category Data science agent): overall 49, evidence grade B, 9 independent performance sources (2 peer-reviewed; competing-author disclosures). Developer results not counted. Methodology: added the Data science agent category. |
| 2026-10-17 | 0028 | Added Perplexity Comet (agentic browser; Browser/web agent): overall 41, evidence grade B-, 4 independent performance sources (1 peer-reviewed; disclosures noted). Trail of Bits audit (vendor-paid) and Perplexity research not counted. Scope note added to the Perplexity entry; its score is unchanged. |
| 2026-10-17 | 0027 | Added Skyvern (open-source; Browser/web agent): Insufficient evidence, 2 independent performance sources (When Bots Take the Bait; WebTrap Park; shared authors disclosed). Web Bench excluded as vendor co-built; Skyvern WebVoyager result not counted. Two CVEs listed with fixes. |
| 2026-10-17 | 0026 | Added STORM / Co-STORM (knowledge-storm library; Research): overall 56, evidence grade B, 7 independent performance sources (5 peer-reviewed). Hosted research preview not covered. |
| 2026-10-17 | 0025 | Added Agent S (open-source framework; new category Computer-use agent): overall 49, evidence grade B, 7 independent performance sources (2 peer-reviewed). Commercial Simular products not covered. Methodology: clarified that 7+ sources without the A conditions receive grade B; added a category list. |
| 2026-10-17 | 0024 | Added Kimi Researcher (Research): Insufficient evidence, 2 independent performance sources (DeepResearch Bench; ICBCBench, with disclosures). Vendor benchmark claims not counted. April 2026 Kimi-wide résumé leak listed under disputes. |
| 2026-10-17 | 0022 | Added Consensus (Research; all modes): overall score 62, evidence grade B, 6 independent peer-reviewed performance sources (2024–2026). Deep Search and Research Agent not separately tested. One abstract-only study, a competitor benchmark and a retracted study listed but not counted. |
| 2026-10-17 | 0021 | Added Claude Research (Research): overall score 58, evidence grade B-, 4 independent performance sources (3 peer-reviewed), all testing 2025 versions; current Claude Opus 5.5 configuration not independently tested. Mind2Web 2 counted with an Amazon/Anthropic investment disclosure. API-model studies listed but not counted. |
| 2026-10-17 | 0020 | Added Manus (Research): Insufficient evidence, 0 independent performance sources for Manus 2.0 (28 Sep 2026, new architecture). Six independent tests of 2025 versions are listed as historical context, not counted. Browser Use: added a competitor disclosure to the CAP source (one author interned at FellouAI, a system evaluated in CAP); score unchanged. |
| 2026-10-17 | 0019 | Rechecked OpenAI Deep Research: score 65 and evidence grade B unchanged. Added a dated version note (February 2026 update; legacy mode removed 26 March 2026; counted tests used 2025 versions; current version not independently tested) and the ShadowLeak security finding (fixed August 2025). |
| 2026-10-17 | 0018 | Added ChatGPT Work (Browser/web agent): Insufficient evidence, 0 independent performance sources for the current product. Its retired predecessors, Operator and ChatGPT agent, are not scored; their independent tests are listed as historical context. Methodology clarification: products that the vendor has retired are not given new scored entries. |
| 2026-10-17 | 0017 | Added Elicit (Research; all tiers, tested mainly on paid tiers): overall score 68, evidence grade A, 7 independent performance sources (6 peer-reviewed; one partly independent). Two abstract-only studies listed but not counted. Search results weak; extraction stronger. |
| 2026-10-17 | 0016 | Added Browser Use, open-source library only (new category: Browser/web agent): overall score 51, evidence grade A, 7 independent performance sources (3 peer-reviewed; 3 are security tests). Results shown as ranges across language models; Browser Use Cloud not scored; tests of 2025 versions counted. |
| 2026-10-17 | 0015 | Added Gemini Deep Research (Research): overall score 63, evidence grade B, 7 independent performance sources. Tests of earlier versions (Gemini 2.5 Pro and 3 Pro) counted, with the current version (3.1 Pro) disclosed; use of a Gemini AI judge in some benchmarks disclosed. |
| 2026-10-16 | 0014 | Design v2: directory sorted by overall score (ties alphabetical) with Insufficient evidence entries listed separately; stacked cards on narrow screens; semantic page structure; Changelog in navigation; entry list in structured data (names and links only); llms.txt. No score, grade, status or entry wording changed. |
| 2026-10-16 | 0013 | Visual design refresh: dark theme by default with automatic light theme, accessible contrast, visible focus styles. Stylesheet only; no score, grade, status or wording changed. |
| 2026-10-16 | 0012 | SEO basics: sitemap, robots.txt, canonical links, page metadata for link previews, a not-found page and an icon. No score, grade, status or wording changed. |
| 2026-10-16 | 0011 | Site polish: “Last reviewed” defined; standard footer links; this changelog; email-only correction requests; Pactum Bloomberg source labeled not reviewed (paywalled); page descriptions, section anchors and consistent site name. No score, grade or status changed. |
| 2026-10-16 | 0010 | Scoring rules v0.1: exact .5 rounds up (permanent); independent performance source rule; evidence grade table; wording cleanups. No score, grade or status changed. |
| 2026-10-16 | 0009 | Added Google DeepMind SAFE: overall 66, grade B-. |
| 2026-10-16 | 0008 | Added Open Interpreter: Insufficient evidence. |
| 2026-10-16 | 0007 | Added Julius AI: Insufficient evidence. |
| 2026-10-16 | 0006 | Added Clay: Insufficient evidence. |
| 2026-10-16 | 0005 | Added Pactum: Insufficient evidence. |
| 2026-10-16 | 0004 | Added Perplexity: overall 54, grade B. Interim rounding note (exact .5 rounds up). |
| 2026-10-16 | 0003 | Added OpenAI Deep Research: overall 65, grade B. |
| 2026-10-16 | 0002 | Added GPT Researcher: overall 53, grade B-. |
| 2026-10-16 | 0001 | Seed list approved; Loki replaced by Google DeepMind SAFE. Scoring rules v0 published. |