Methodology
Scoring rules v0.1, updated 2026-10-17 (Decisions 0010 and 0018).
Compare: Claude Code and OpenAI Codex compared · Coding agents compared · Research agents compared
How this directory works
Counted independent evidence
An overall score is given only when there are at least 3 independent performance sources: benchmarks, research studies, documented hands-on tests, security or safety evaluations or audits that report their method and results. Independent means not produced, funded or co-authored by the vendor or maintainer, and not published by a competing product to promote itself. A source counts only if it tested the product, version and scope covered by the entry, and each study counts once. Reviews, commentary, press coverage without original testing and product descriptions are listed as other sources; they can inform factor scores but do not count toward the 3.
Separate factor scores
Each scored entry shows four factors, each scored 0–100: completion rate (30%), disputes (20%), adherence to limits (25%) and evidence quality (25%), together with the weighted overall score. The evidence grade (A, B or B-) describes the amount and type of independent evidence behind a score, not how well the agent performs.
Open data
The same entries are published as agents.json, agents.csv and schema.json under CC BY 4.0, generated from the same source files as the pages. agents.json lists every cited source with its kind, access date and whether it was counted. Changes are recorded in the changelog and the update feeds, and the decision records are in the public repository.
No ranking or endorsement
Scored entries are listed by overall score, but order is not a recommendation or a ranking of quality. Scores are informational, rule-based summaries of public evidence as of each entry's last-reviewed date; they are not a guarantee, endorsement or certification. Entries are not removed, added or re-scored for payment.
Each factor is scored 0–100 and shown separately, together with a weighted overall score.
| Factor | Weight | How it is judged |
|---|---|---|
| Completion rate | 30% | Share of tasks completed as described, per public evidence. |
| Disputes | 20% | Documented complaints, legal or regulatory actions and security advisories concerning the agent; fewer = higher score. Pending cases are recorded as allegations, not findings. Errors in the agent's output are scored under Completion rate. |
| Adherence to limits | 25% | Stays within stated scope, permissions and safety limits. |
| Evidence quality | 25% | Independent third-party sources rate high; vendor claims low; anonymous anecdotes lowest. |
Rounding
The overall score is calculated from the unrounded factor weights and then rounded to the nearest whole number. A result of exactly .5 is rounded up (for example, 53.5 is shown as 54). Factor scores are whole numbers and are not rounded further.
Insufficient evidence
An overall score is given only when there are at least 3 independent performance sources. Otherwise the entry is labeled "Insufficient evidence" and no overall score is given; factor notes may still be shown.
Performance source: a source that tests or measures the agent's performance, behaviour or outcomes, such as a benchmark, research study, documented hands-on test, security or safety evaluation, or audit, and reports its method and results. Product descriptions, reviews, user-review aggregates, press coverage without original testing, commentary, court filings and vulnerability listings are recorded as other sources. They can inform factor scores but do not count toward the 3.
Independent: not produced, funded or co-authored by the vendor/maintainer or by a customer's staff, and not published by a competing product to promote itself. Studies by researchers who propose a competing research method may count; this is disclosed on the entry.
Counting: each study counts once, however many versions or links it has. A source counts only if it tested the product, version and scope covered by the entry. A source whose independence is only partial (for example, relying on data supplied by a customer) is labeled as such.
Retired products: Products that the vendor has retired are not given new scored entries; independent tests of them may be listed as historical context. (Clarification, Decision 0018.)
Evidence grade
The evidence grade describes the amount and type of independent evidence behind a score, not how well the agent performs. It is set by this table, using independent performance sources as defined above:
| Grade | Independent performance sources |
|---|---|
| A | 7 or more, at least 3 peer-reviewed, at least 1 testing the current version within the last 12 months |
| B | 5–6, or 7+ without meeting the A conditions |
| B- | 3–4 |
| Insufficient evidence | fewer than 3 (no grade and no overall score) |
Clarification (decision 0025): an entry with 7 or more counted sources but fewer than 3 peer-reviewed (or without a recent current-version test) receives grade B.
Categories
Research; Verification; Browser/web agent; Computer-use agent; Data science agent; Coding agent; Data enrichment; Negotiation; Simple analysis.
Coding agent: an agent that works in a software repository or development environment to write, edit, run and test code over multiple steps, with limited step-by-step supervision (for example, resolving an issue, making a multi-file change, or fixing failing tests).
- An agent that operates a desktop or apps through the screen, even if it sometimes opens a code editor, is a Computer-use agent.
- An agent that writes code mainly to analyse data or build models and report results is a Data science agent.
- An agent that runs code locally for general tasks, without a focus on working in a repository, is Simple analysis.
- Inline code completion or chat-only code suggestions without autonomous multi-step execution are out of scope.
- A product that spans categories is placed in the one that matches its main documented use.
The evidence-quality factor score should be consistent with the grade (as a guide: A ≥ 85, B 70–84, B- 55–69). If they diverge, the decision packet explains why.
Directory order
Entries with an overall score are listed from highest to lowest overall score. Equal scores are listed alphabetically by entry name (case-insensitive). Entries labeled Insufficient evidence are listed separately, alphabetically. Order is not a recommendation.
Last reviewed
The date (America/New_York) an approved decision packet last re-checked an entry's sources and scores against the current rules. Wording, formatting or link-label fixes do not change it. A rules change that leaves an entry's outcome unchanged (e.g. Decision 0010) does not change it either. It is not a guarantee that information is current.
Corrections
If you believe an entry contains a factual error or misses a relevant public source, email trustkeeperace@gmail.com with the entry name, what you believe is wrong, and links to public sources.
- Requests are reviewed under the same published rules as every entry. A request may or may not lead to a change; any change is made through an approved decision and listed in the changelog.
- We do not promise a response or a response time.
- Entries are not removed, added or re-scored for payment, and we do not accept payment for any listing or change.
- Please do not send confidential or personal information. Only public sources can be used.
- Sending a request does not create any agreement or obligation.
Privacy and measurement
This site does not use cookies, advertising, or cross-site tracking, and it does not sell or share visitor data. It contains no tracking scripts.
To understand whether the Clearinghouse is useful, we look at aggregate counts only:
- how the site appears in Google Search, using Google Search Console (Google’s Privacy Policy and Terms of Service apply to that service);
- request totals, most-requested pages and AI-crawler totals from our hosting provider’s built-in request statistics (Vercel; see its Privacy Policy).
We record only monthly totals, not information about individual visitors. Traffic figures are approximate and are never used in agent scores or grades. Questions: trustkeeperace@gmail.com.
Data and license
The directory data is available as JSON and CSV on For agents / data. The data and site text are licensed under CC BY 4.0.
Citing or quoting scores: How to cite.
Disclaimer
Informational only; not a guarantee, endorsement, or certification. Scores reflect public sources and the stated rules at the time of review, and do not represent verification beyond what is explicitly stated. No payments are handled. The Clearinghouse is not responsible for any agent's outcomes.