FacTool (open-source)

Citation check incomplete — recheck later.

Summary

FacTool is open research code from GAIR-NLP that splits text into claims, searches the web for evidence and labels each claim true or false. Seventeen independent sources ran it (thirteen peer-reviewed), almost all as a comparison system in papers proposing alternative methods. On standard claim sets it labels true claims fairly well (F1 about 0.60-0.88) and false claims less well (about 0.46-0.64), and two studies report it as slower or costlier than newer checkers. One study found it overstated factual precision on long answers. The code has not changed since 2023; an unfixed medium-severity code-injection flaw (CVE-2025-4261) affects its math mode. Informational only; not a guarantee, endorsement, or certification.

Overall score: 55 · Evidence grade: B · Independent sources: 17 performance (13 peer-reviewed), 10 other

Evidence grade describes the amount and type of evidence, not the agent's quality.

Grade note: Grade B: 17 counted sources (13 peer-reviewed), but no test within 12 months states the FacTool version or commit tested. The code has not changed since about October 2023, so recent tests very likely ran the only version; under the current rule a stated version is still required for A (see Decision 0084, D7). (see methodology)

Factor breakdown

0.30 × 50 + 0.20 × 55 + 0.25 × 40 + 0.25 × 75 = 54.75, rounded to 55 (rounding rule).

Scope

The open-source FacTool framework (GitHub GAIR-NLP/factool and PyPI factool 0.1.x, 2023; code unchanged since about October 2023), which splits text into claims, searches for evidence and labels each claim, with modes for knowledge-based QA, code, math and scientific literature review. Tests that run FacTool's pipeline with a different language model count as tests of FacTool. Results for a language model alone (0039), Halu-J, and reuse of FacTool's datasets or prompts are not covered. Results depend heavily on the language model and search settings used.

How sources were counted

The FacTool paper, repository and package are the developers' own material and are not counted. Papers that run FacTool as a comparison system count, with their competing methods disclosed; tests of only the verification step count as component tests. Five counted papers come from one connected group of MBZUAI-affiliated authors and share some result rows. Papers that repeat earlier FacTool figures without a new run, use FacTool only as a metric, or adapt its prompts are listed as other. One study with a FacTool author is excluded.

Security and disputes

CVE-2025-4261 (published 5 May 2025; CVSS 4.0 score 4.8, Medium; local attack) describes code injection through exec() in FacTool's math mode, which runs model-generated code. It was reported in GitHub issue #50 in April 2025; no maintainer reply or fix was found. FacTool's code has not changed since about October 2023. No legal or regulatory action was found.

Uncertainties

• Nearly every counted study proposes a competing method and reports FacTool as a baseline.
• Five counted studies come from one connected group of MBZUAI-affiliated authors (Factcheck-Bench, OpenFactCheck, Loki, FIRE, UrduFactCheck); some reuse each other's FacTool rows, so they are not independent samples.
• Studies use different language models (GPT-3.5 to GPT-4.1, LLaMA, Qwen) and search settings; figures describe FacTool with that model.
• Three counted studies test only the verification step (Factcheck-Bench, OpenFactCheck, REVEAL); others apply FacTool outside its original tasks (summaries in FIZZ, agent trajectories in Phantom Merge, Urdu translations in UrduFactCheck, mitigation in Rowen).
• No study states a FacTool version or commit; the code has not changed since about October 2023, so this is the only version.
• One study with a FacTool author (Knowledge-Centric Hallucination Detection / RefChecker) is excluded under 0031.
• Citation check incomplete — recheck later (OpenAlex full text used; Semantic Scholar rate-limited and the arXiv API unreliable; 29 citing works could not be downloaded).

Sources

Independent performance sources (counted)

• Factcheck-Bench (MBZUAI and others; Findings of EMNLP 2024, arXiv 2311.09000): FacTool's verification step on 678 human-annotated claims; F1 for true claims 0.84 and for false claims 0.47 with an LLaMA backend (0.81 and 0.46 with GPT-3.5). Component test. Disclosure: MBZUAI-affiliated authors who propose or maintain competing fact-checkers (Factcheck-GPT, OpenFactCheck, Loki, FIRE); none is a FacTool author. Setup: verification step only, Serper web evidence — accessed 2026-10-19
• OpenFactCheck (MBZUAI, Monash, Sofia; COLING 2025, arXiv 2405.05583): FacTool checker on Factcheck-Bench (rows shared with Factcheck-Bench), FacTool-QA and FELM-WK; reported cost about $2.5 search plus $12.2 GPT-3.5 and 0.49 hours for 765 claims. Component test. Disclosure: MBZUAI-affiliated authors who propose or maintain competing fact-checkers (Factcheck-GPT, OpenFactCheck, Loki, FIRE); none is a FacTool author. Setup: verification step, GPT-3.5 and Serper — accessed 2026-10-19
• Loki (LibrAI, MBZUAI, Monash, Melbourne; COLING 2025 System Demonstrations, arXiv 2410.01794): FacTool with GPT-4o on Factcheck-Bench (true-claim F1 0.71, false-claim F1 0.61) and FacTool-QA (0.78 and 0.56); 12.38 seconds per sample against 8.32 for Loki. Earlier rows reused from OpenFactCheck. Disclosure: MBZUAI-affiliated authors who propose or maintain competing fact-checkers (Factcheck-GPT, OpenFactCheck, Loki, FIRE); none is a FacTool author. Setup: GPT-4o and Serper — accessed 2026-10-19
• FIRE (MBZUAI, TU Darmstadt; Findings of NAACL 2025, arXiv 2411.00784): FacTool with GPT-4o on FacTool-QA true-claim F1 0.84 and false-claim F1 0.58; FELM-WK 0.60 and 0.64; BingCheck 0.68 and 0.56; reported cost $24.76 model plus $3.67 search against $3.35 plus $0.60 for FIRE. Disclosure: MBZUAI-affiliated authors who propose or maintain competing fact-checkers (Factcheck-GPT, OpenFactCheck, Loki, FIRE); none is a FacTool author. Setup: GPT-4o and GPT-4o-mini with Serper — accessed 2026-10-19
• UrduFactCheck (MBZUAI and others; Findings of EMNLP 2025, arXiv 2505.15063): FacTool run on Urdu translations of FacTool-QA and BingCheck as a comparison system. Translated-data adaptation. Disclosure: MBZUAI-affiliated authors who propose or maintain competing fact-checkers (Factcheck-GPT, OpenFactCheck, Loki, FIRE); none is a FacTool author. Setup: GPT-4o and GPT-4o-mini — accessed 2026-10-19
• G2LDetect (Zhejiang University; AAAI 2025): FacTool baseline on HaluEval-Summary, QAGS, FEVER and SciFact; for example F1 about 70.9 on HaluEval-Summary with GPT-4; authors propose an alternative method. Setup: Llama3-8B, ChatGPT and GPT-4 backends — accessed 2026-10-19
• FIZZ (EMNLP 2024): FacTool baseline for factual inconsistency in summaries (AggreFact); balanced accuracy 49.3-72.7 across subsets, below the authors' method. Use outside FacTool's original task design; authors propose an alternative method — accessed 2026-10-19
• REVEAL, A Chain-of-Thought Is as Strong as Its Weakest Link (Google Research, Tel Aviv University and others; ACL 2024): FacTool macro-F1 71.1 on step attribution (2-class), with evidence supplied from the dataset instead of FacTool's own search. Component test. Setup: verification step with dataset evidence inserted — accessed 2026-10-19
• FaStFact (Findings of EMNLP 2025, arXiv 2510.12839): FacTool scored F1@K' 0.987 against a ground truth of 0.792 on FaStfact-Bench (overestimate 0.195, the largest gap of five pipelines) and differed by 9.09 claims per response from the annotated count; authors propose an alternative method. Setup: knowledge-based QA setting — accessed 2026-10-19
• Rowen (Chinese Academy of Sciences; SIGIR-AP 2025, arXiv 2402.10612): FacTool as a hallucination-mitigation baseline on TruthfulQA and StrategyQA scored below the authors' method and used 12.5 and 11.6 retrieval calls per question (Rowen 1.5 and 0.5); authors propose an alternative method — accessed 2026-10-19
• EMULATE (National University of Singapore; FEVER 2025 workshop proceedings): FacTool with GPT-4.1 on FacTool-KBQA (true-claim F1 0.87), BingCheck (0.88) and Factcheck-Bench (0.82), with false-claim precision 0.52-0.59; authors propose an alternative method. Setup: GPT-4.1 (gpt-4.1-2025-04-14) — accessed 2026-10-19
• JointCQ (Peking University, Huawei; Findings of ACL 2026, arXiv 2510.19310): FacTool with GPT-4.1 on ANAH English and Chinese hallucination detection; the authors report their method about 5-8 points higher in accuracy. Setup: GPT-4.1 backend — accessed 2026-10-19
• LEAP, Can a Small Model Learn to Look Before It Leaps? (LNCS 2026, arXiv 2511.05854): FacTool, SAFE and FIRE run as tool-augmented baselines with small open models on three hallucination-detection datasets; FacTool accuracy about 47-62%, below FIRE in two of three result blocks and above it in the third; authors propose an alternative method. Setup: Qwen2.5-7B, Llama3.1-8B and other small backends — accessed 2026-10-19
• Re-Ex (Yonsei University; ICLR 2024 workshop, non-archival): FacTool with the official code on FactPrompt and WiCE; FactPrompt detection F1 52.6 (GPT-3.5) and 57.1 (GPT-4), 20.0-26.2 seconds and 3.1-4.1K tokens per response; authors propose an alternative method. Setup: GPT-3.5 and GPT-4 — accessed 2026-10-19
• PASS-FC (Trip.com Group; preprint): FacTool baseline on FacTool-QA, FELM-WK and Factcheck-GPT with default settings; FacTool-QA F1 62.5, below the author's method; authors propose an alternative method. Setup: GPT-3.5-turbo-0613, default settings — accessed 2026-10-19
• CoVer (preprint Sep 2026): FacTool with GPT-4o (two search queries, ten results each) on seven claim sets with conflicting evidence; FacTool accuracy 51.0-81.5% (51.0% on FEVER, the lowest of six systems; 80.0% on CONFACT HumC, against 88.4% for the authors' method); authors propose an alternative method. Setup: GPT-4o backend — accessed 2026-10-19
• Phantom Merge (preprint Oct 2026): FacTool as a detection baseline for agent claims that misreport which tool result was used; AUROC 0.679 against 0.582 for FActScore and 0.499 for RARR, below the authors' method; task outside FacTool's original design — accessed 2026-10-19

Other sources (not counted)

• UFO (Renmin University; preprint): uses FacTool's pipeline as a reference evaluation metric for other models. FacTool used as a tool, not tested — accessed 2026-10-19
• MERMAID (preprint Jan 2026): its FacTool GPT-4o row on FacTool-QA (0.84 and 0.58) matches FIRE's published figures and follows FIRE's setup; a separate run is not shown — accessed 2026-10-19
• Markov chain multi-agent debate (ICASSP 2025): the FacTool rows match the figures in the FacTool paper; a separate run is not shown — accessed 2026-10-19
• FactSearch (ACL 2026 demonstrations): FacTool results reproduced from the FacTool paper, not rerun — accessed 2026-10-19
• OpenFactCheck demonstration paper (EMNLP 2024 System Demonstrations; MBZUAI and others): leaderboard of the same FacTool results as the COLING 2025 paper (accuracy 63%, $12.2, 0.49 hours). Duplicate figures, listed once — accessed 2026-10-19
• LEMMA (preprint 2024): FacTool with GPT-4 given image summaries as a multimodal misinformation baseline on Fakeddit and a rumour set. Adapted pipeline outside FacTool's text-only design — accessed 2026-10-19
• GAMED (preprint Dec 2024): FacTool accuracy 0.506 on Fakeddit, apparently taken from earlier work; a separate run is not shown — accessed 2026-10-19
• Multi-agent credibility scoring (Scientific Reports 2026): adapts FacTool prompts in its own system; FacTool itself is not run — accessed 2026-10-19
• CVE-2025-4261 (VulDB, published 5 May 2025; CVSS 4.0 4.8 Medium; local attack): code injection through exec() in factool/math/tool.py, up to commit 3f3914b; reported in GitHub issue #50 (21 Apr 2025) with no maintainer reply; no fix found; NVD status Deferred. No GitHub security advisory — accessed 2026-10-19
• GitHub issue #48 (Oct 2024): a user reports errors running the bundled example; no fix found. Single user report — accessed 2026-10-19

Not counted

• FacTool paper (GAIR-NLP, SJTU, CMU and others; arXiv 2307.13528): the developers' own results across knowledge-based QA, code, math and scientific literature review. Not counted — accessed 2026-10-19
• GitHub repository (Apache-2.0): FacTool code commits end about October 2023; last push 19 August 2024 (Halu-J material); no GitHub releases; not archived; 23 open issues. Not counted — accessed 2026-10-19
• PyPI package factool: latest release 0.1.3 (14 August 2023); pins openai==0.27.8. Not counted — accessed 2026-10-19
• Knowledge-Centric Hallucination Detection / RefChecker (Amazon AWS AI and others; EMNLP 2024, preprint arXiv 2405.14486): runs FacTool with GPT-4 on KnowHalBench, but Pengfei Liu, a FacTool author, is a co-author. Excluded under 0031 (shared author) — accessed 2026-10-19
• Halu-J (GAIR-NLP; preprint 2024): a separate judge model by FacTool authors kept in the same repository. Outside this entry — accessed 2026-10-19
• Studies that only reuse FacTool's datasets (for example FacTool-QA) or test a language model alone (out of scope).

Decision record: 0084.