Agent Reputation & TrustClearinghouse

AIDE (Weco AI, open-source)

Category
Data science agent
Type
Open-source machine-learning engineering agent (MIT licence) from Weco AI; PyPI aideml, GitHub WecoAI/aideml; runs with a language model chosen by the user
Source
Project repository
Last reviewed
2026-10-17 (what this means)

Citation check incomplete — recheck later.

Summary

AIDE is an open-source agent from Weco AI that tackles machine-learning tasks, such as Kaggle competitions, by writing, running and refining candidate solutions in a search tree. It is run with a language model chosen by the user. Thirteen independent studies that ran it were found (six peer-reviewed); most of them propose a competing method and use AIDE as a baseline. Results depend heavily on the model and task. On OpenAI's MLE-bench, o1-preview with AIDE earned a medal in 16.9% of 75 Kaggle competitions, and GPT-4o with AIDE did better than with two general-purpose agents (8.7% vs 0.8% and 4.4%). Several newer methods report better results than AIDE (for example 55% vs 66% of teams beaten on MLE-bench Lite), and one benchmark found it behind simpler search strategies. Studies of how it works found that 85-88% of its pipelines reported only a single evaluation metric, and that its pipelines were well below expert solutions on fairness. No CVE or legal action was found. No test from the last 12 months states the AIDE version. Informational only; not a guarantee, endorsement, or certification.

Overall score: 60 · Evidence grade: B · Independent sources: 13 performance (6 peer-reviewed), 6 other

Evidence grade describes the amount and type of evidence, not the agent's quality. Grade B rather than A: no independent test from the last 12 months states the AIDE version (see methodology).

Factor breakdown

Factor breakdown
FactorWeightScoreNote
Completion rate30%50MLE-bench (75 Kaggle competitions, 24 h): medal in 16.9% with o1-preview; GPT-4o 8.7% (MLAB 0.8%, OpenHands 4.4%); Claude 3.5 Sonnet and Llama 3.1 405B lower. TimeSeriesGym (GPT-4.1): valid submissions 57.3%, reasonable 12.5%; better than OpenHands with the same model (66.7% vs 44.4% valid on a subset). ML-Dev-Bench (30 ML development tasks): 17% success with GPT-4o (OpenHands + Claude 3.5 Sonnet 50%). CTRL-ALT-DECEIT: just above the human median on its MLE-bench tasks, ahead of a ReAct agent. MARS controlled runs (MLE-bench Lite): any medal 36.4% (Gemini 2.5 Pro) and 53.0% (Gemini 3 Pro Preview), below AIRA-dojo and MARS. ArchPilot (all 75 MLE-bench tasks, same budget): valid submissions 78.7%; beat 55% of teams on MLE-bench Lite vs 66% for ArchPilot. CO-Bench: below simpler strategies such as BestOfN on most metrics, with higher valid-solution rates. InnoGym: lowest of three frameworks on its novelty measure. Simple Baselines: a simple sequential search matched or beat AIDE on 6 of 10 competitions. SPIO (tabular tasks, GPT-4o): close to other baselines, below SPIO. Can LLM Agents Discover?: attempt success 45.0% (GPT-5) and 26.2% (Qwen3-32B).
Disputes20%70No CVE, legal or regulatory action found. MLE-bench's authors note that Weco's own April 2024 report (beating more than half of human competitors) used easier competitions than MLE-bench. The project is maintained: v0.2.2 (November 2025), with dependency security fixes merged in October 2026 (issue #100, a path traversal in the datasets dependency).
Adherence to limits25%45The Silent Failures in Agentic Data Science (CWI, preprint Oct 2026): AIDE removed missing validation splits but reported only one evaluation metric in 85-88% of pipelines, and with a small model its use of validation splits increased some leakage failure modes. Be Fair! (preprint 2026): on a dermatology task, AIDE had higher AUC and a smaller fairness gap than MLZero, but all agent pipelines were at least 10 AUC points below expert solutions and dominated on fairness. CTRL-ALT-DECEIT (NeurIPS 2025): when instructed to, AIDE agents sometimes sabotaged code or underperformed without being caught, although the monitors were overall highly effective; this tests capability under instruction, not unprompted behaviour. No independent study of unrequested actions in normal use was found.
Evidence quality25%80Thirteen independent sources, six peer-reviewed (ICLR 2025, NeurIPS 2025, AAAI 2026, ICLR 2026, COLM 2026, ACL 2026). Most are competing-method papers that use AIDE as a baseline, and several used forks adapted for their benchmark. No test states a release version; the only stated build is a May 2025 fork (TimeSeriesGym). No versioned test from the last 12 months, so grade B (0025 clarification; SWE-agent precedent).

0.30 × 50 + 0.20 × 70 + 0.25 × 45 + 0.25 × 80 = 60.25, rounded to 60 (rounding rule).

Scope

The open-source AIDE agent (aideml), which searches a tree of candidate Python solutions for a machine-learning task, from v0.1.x (April 2024) through v0.2.2 (5 November 2025) and the main branch (last commit 16 October 2026). Re-implementations of AIDE inside other frameworks (for example Meta's AIRA-dojo) and Weco's commercial platform are not covered. Results depend heavily on the language model, task and compute budget.

How sources were counted

Weco AI's own reports are not counted. Studies with AIDE developers as co-authors are excluded (FML-bench). Re-implementations of AIDE inside other frameworks, and papers that only reuse published AIDE numbers or trajectories, are listed as other sources. Most counted studies propose a competing method and use AIDE as a baseline; they count with disclosure. AutoKaggle, a separate project from different developers, has its own entry.

Security

AIDE runs code written by the model on the user's machine or in a container. No CVE for AIDE was found. A vulnerable dependency (datasets) was patched in October 2026. No legal or regulatory action was found.

Uncertainties

Sources

Independent performance sources (counted)

Other sources (not counted)

Not counted

Decision record: 0042.