STORM / Co-STORM (knowledge-storm library)
- Category
- Research
- Type
- open source (MIT), PyPI knowledge-storm; Stanford hosted research preview (storm.genie.stanford.edu) not covered
- Source
- Project repository
- Last reviewed
- 2026-10-17 (what this means)
Citation check incomplete — recheck later.
Summary
STORM is an open-source research system from Stanford's OVAL lab. It writes Wikipedia-style articles with citations by searching the web, simulating expert conversations and drafting an outline. Co-STORM adds a mode where a human takes part in the conversation. Results depend on the language model and search engine used. In seven independent evaluations (five peer-reviewed: EMNLP 2025 x2, ACL 2025, ACL 2026, ACL 2026 Findings), STORM produced complete cited articles but usually ranked below newer research systems. About 66-77% of citations supported the claims in one Wikipedia-generation study, and it scored 32.8-35.3 on DeepResearch Bench (50 = parity with the reference report). Co-STORM was tested in three of these studies with similar or lower results. The code has not changed since September 2025, and a July 2025 security report about disabled certificate checking remains unanswered. Overall score 56. Informational only; not a guarantee, endorsement, or certification.
Overall score: 56 · Evidence grade: B · Independent sources: 7 performance, 2 other
Evidence grade describes the amount and type of evidence, not the agent's quality.
Grade note: Grade B rather than A: 7 counted sources (5 peer-reviewed), but none from the last 12 months states that it tested the current library version; only WriteHERE states a version (v1.1.0, 2025). (see methodology)
Factor breakdown
| Factor | Weight | Score | Note |
|---|---|---|---|
| Completion rate | 30% | STORM reliably produces full cited articles, but quality trails newer systems. LLM-judge rubric scores: 4.30-4.76 of 5 with GPT-4o and Claude 3.5 (WriteHERE); 2.42-4.34 of 5 across backbones (OmniThink). Relative score 0.32-0.34 where 0.50 means parity with a human or Gemini Deep Research reference (CogGen). DeepResearch Bench RACE 32.81 (Qwen3-32B) and 35.26 (DeepSeek-V3.2), lowest of the multi-agent baselines (ScaffoldAgent). Answer coverage 72.0%, lowest of six systems (MySQA). DeepScholar-Bench geometric mean .073 (OpenAI Deep Research .309; human .782). Co-STORM: similar or lower in OmniThink, WriteHERE and CogGen. Judgment call. | |
| Disputes | 20% | No legal, regulatory or Wikipedia-community action found. A public GitHub issue (#375, 1 July 2025) reported an improper certificate validation problem and asked for a private disclosure channel; there has been no maintainer reply. The current code downloads web pages with certificate verification disabled (verify=False). No CVE was found. The project is active but dormant: no code changes since 30 Sep 2025 (last release 1.1.1), and 111 issues are open. Judgment call. | |
| Adherence to limits | 25% | Citation grounding is partial. Citation recall 77.49% / 67.72% and precision 75.71% / 66.50% on FreshWiki / WikiStart (MOG). Citation precision 73.3% and recall 64.7% (MySQA). DeepScholar-Bench citation precision .238 and claim coverage .586 with Llama-4. DeepResearch Bench citation accuracy 36.90% / 38.59% (ScaffoldAgent). Searches are limited to the configured retriever; no safety or prompt-injection tests were found. Judgment call. | |
| Evidence quality | 25% | Seven independent performance sources, five peer-reviewed, three published in 2026. All are by authors of competing systems or benchmarks (disclosed). Most use LLM judges and relative scores. None states the library version except WriteHERE (v1.1.0, 2025), so the A condition of a current-version test within 12 months is not met (grade B under the 0025 clarification). Judgment call. |
0.30 × 50 + 0.20 × 55 + 0.25 × 45 + 0.25 × 75 = 56.00, so 56 (no rounding needed; rounding rule).
Scope
This entry covers the open-source knowledge-storm library from Stanford’s OVAL lab (MIT licence), run with a language model and search engine chosen by the user. It includes both modes: STORM (automatic article writing) and Co-STORM (a human takes part in the research conversation); Co-STORM results are labelled below. The hosted research preview (storm.genie.stanford.edu) is not covered; no independent test of it was found.
Version and status note
Versions 1.0.0 (September 2024) to 1.1.1 (September 2025) are treated as one continuing library. The 2024 NAACL code branch is historical context; one study found v1.1.0 slightly stronger than it. The project is active but dormant: it is not archived and the research preview is online, but there have been no code changes since 30 Sep 2025.
Independent results by model
Most results are LLM-judge or relative scores comparing STORM with other systems, not success rates. The developers’ own results are not counted.
| Mode | Study | Result |
|---|---|---|
| STORM | WriteHERE (v1.1.0), judge scores of 5 | GPT-4o 4.30–4.76; Claude 3.5 Sonnet 4.40–4.66 |
| Co-STORM | WriteHERE, judge scores of 5 | GPT-4o 4.02–4.36; Claude 3.5 Sonnet 3.46–3.87 |
| STORM | OmniThink (WildSeek), judge scores of 5 | GPT-4o 3.80–4.34; other backbones as low as 2.42 |
| STORM | MOG, citation recall / precision | 77.49 / 75.71% (FreshWiki); 67.72 / 66.50% (WikiStart) |
| STORM / Co-STORM | CogGen, relative score (0.50 = parity) | STORM 0.3205 / 0.3351; Co-STORM 0.3227 / 0.3147 |
| STORM | MySQA, answer coverage / citation precision | 72.0% / 73.3% |
| STORM | DeepResearch Bench via ScaffoldAgent, RACE | 32.81 (Qwen3-32B); 35.26 (DeepSeek-V3.2) |
| STORM | DeepScholar-Bench (Llama-4), geometric mean | .073 (OpenAI Deep Research .309; human .782) |
Security and disputes
- GitHub issue #375 (1 July 2025) reported an improper certificate validation problem and asked for a private disclosure channel. There has been no maintainer reply.
- The current code (commit fb951af) downloads web pages with certificate verification disabled (
verify=False). - No CVE, security advisory, legal or regulatory action, or Wikipedia-community action was found.
Uncertainties
- Most independent results use LLM judges and relative scores, so they show how STORM compares with other systems rather than an absolute success rate.
- Results depend on the language model and search engine. Most studies used GPT-4o-era or open models.
- Only one study states the library version; the library has not changed since September 2025.
- Co-STORM appears in three studies only; there is no independent test of its human-in-the-loop mode with real users.
- The hosted research preview is not covered and was not independently tested.
- Citation check incomplete — recheck later (0052: Semantic Scholar unavailable; OpenAlex title screen only).
Sources
Independent performance sources (counted)
- WriteHERE, "Beyond Outlining" (EMNLP 2025; KAUST and others; arXiv 2503.08275): reran STORM v1.1.0 with Serp/Bing search, report-quality judge scores of 4.76/4.58/4.30/4.32 (relevance/breadth/depth/novelty, of 5) with GPT-4o and 4.66/4.63/4.40/4.41 with Claude 3.5 Sonnet. Co-STORM 4.36/4.22/4.02/4.17 and 3.87/3.56/3.46/3.82. v1.1.0 was slightly stronger than the 2024 NAACL branch. Competing-method disclosure (WriteHERE scored higher) — accessed 2026-10-17
- OmniThink (EMNLP 2025; Zhejiang University / Alibaba Tongyi; arXiv 2501.09751): reproduced STORM and Co-STORM on WildSeek (100 topics). STORM GPT-4o rubric scores 4.34/4.21/4.21/3.80 of 5, knowledge density 19.33; lower with other backbones (as low as 2.42). Competing-method disclosure — accessed 2026-10-17
- MOG, "Hierarchical Memory Organization for Wikipedia Generation" (ACL 2025; Peking University / Huawei; arXiv 2506.23393): STORM citation recall/precision 77.49/75.71% on FreshWiki and 67.72/66.50% on WikiStart; entity recall 13.37%/20.06%. MOG scored higher on most measures. Competing-method disclosure — accessed 2026-10-17
- CogGen (Findings of ACL 2026; Nanjing University; arXiv 2604.17072): relative scores against a reference (0.50 = parity). STORM 0.3205 (OWID, human reference) and 0.3351 (WildSeek, Gemini Deep Research reference); Co-STORM 0.3227 / 0.3147. Competing-method disclosure — accessed 2026-10-17
- MySQA, "Language Models Don't Know What You Want" (ACL 2026; Allen Institute for AI / UMD / Cornell; arXiv 2603.16120): STORM answer coverage 72.0%, answer precision 92.2%, citation precision 73.3%, citation recall 64.7%, action adherence 74.4%; lowest coverage of six systems. Disclosure: Ai2 builds the competing ScholarQA system — accessed 2026-10-17
- ScaffoldAgent (preprint, June 2026; Peking University): STORM on DeepResearch Bench, RACE 32.81 (Qwen3-32B) and 35.26 (DeepSeek-V3.2), lowest of the multi-agent baselines; FACT effective citations 15.18 / 17.42 and citation accuracy 36.90% / 38.59%. Competing-method disclosure — accessed 2026-10-17
- DeepScholar-Bench (preprint; Stanford Guestrin group and UC Berkeley, not the OVAL lab): STORM with Llama-4-Scout, arXiv-only retrieval, organization .119, nugget coverage .183, citation precision .238, claim coverage .586, geometric mean .073 (OpenAI Deep Research .309; human exemplars .782). Disclosure: same university as the developers; authors propose the competing DeepScholar-ref pipeline — accessed 2026-10-17
Other sources (not counted)
- Public GitHub issue #375 (1 July 2025): reports an improper certificate validation problem in the default configuration and asks for a private disclosure channel; no maintainer reply as of 17 Oct 2026 — accessed 2026-10-17
- Source at commit fb951af (30 Sept 2025, latest): WebPageHelper creates its HTTP client with certificate verification disabled (verify=False) for downloading web pages — accessed 2026-10-17
Not counted
- STORM paper (Stanford OVAL; NAACL 2024): developer-run FreshWiki results and a study with 10 Wikipedia editors (7 of 10 found tone "emotional" or "unneutral"). Not counted — accessed 2026-10-17
- Co-STORM paper (Stanford OVAL; EMNLP 2024): developer-run WildSeek results and user study. Not counted — accessed 2026-10-17
- DeFine (dataset paper): STORM figures are taken from the STORM paper. Not counted — accessed 2026-10-17
- Studies that mention STORM without testing it (e.g. DeepResearch Bench’s own paper, Rao et al.’s reference-hallucination study, DREAM, AutoSurvey, SurveyX, WebWeaver), and FrankenReport, which modifies STORM rather than evaluating it.
Decision record: 0026.