Google DeepMind SAFE (Search-Augmented Factuality Evaluator)
- Category
- Verification
- Type
- research code
- Source
- Project page
- Last reviewed
- 2026-10-16 (what this means)
Citation check incomplete — recheck later.
Summary
SAFE (Search-Augmented Factuality Evaluator) is open research code from Google DeepMind, described in a NeurIPS 2024 paper, that uses a language model and Google Search results to rate individual facts in long-form text. The authors report 72% agreement with crowdsourced annotators. Four independent peer-reviewed studies report agreement with human annotators of about 84% on biography facts and strong correlation on factual labels, alongside weaker results for its claim extraction (including many incomplete extracted facts); these studies were conducted by researchers proposing alternative methods. Commentators have questioned the authors' comparison with crowdworkers rather than expert fact-checkers. Overall score 66. Informational only; not a guarantee, endorsement, or certification.
Overall score: 66 · Evidence grade: B- · Independent sources: 4 performance, 2 other
Factor breakdown
| Factor | Weight | Score | Note |
|---|---|---|---|
| Completion rate | 30% | Independent agreement and correlation are good on factual labels (84.48% on 786 biography facts, Zhao et al.; response-level Pearson 0.95, FactBench), weaker on other labels (0.60) and for claim extraction (VeriScore). VeriFact (EMNLP 2025) found 56.7% of SAFE-extracted facts incomplete on a deliberately difficult annotated set. Judgment call. | |
| Disputes | 20% | No legal or regulatory disputes found. Public critique of the "superhuman" framing (comparison with crowdworkers, not expert fact-checkers). Judgment call. | |
| Adherence to limits | 25% | Clear licences and stated limitations; maintained research code. Independently measured cost ($0.49 per response) is higher than the authors' $0.19, though setups differ. Judgment call. | |
| Evidence quality | 25% | Four peer-reviewed independent sources, but small samples, mainly biographies and chat prompts, and each study promotes its own alternative method. Judgment call. |
Overall = 0.30×65 + 0.20×80 + 0.25×60 + 0.25×60 = 65.5, shown as 66 (exact .5 rounds up). See the methodology.
Uncertainties
- Results depend on which language model SAFE runs with and on the Serper search API.
- Cost figures across studies are not directly comparable.
- All four independent studies come from researchers proposing competing evaluators.
- No independent study using expert fact-checkers was found.
- The authors' 72% and 76% figures are their own claims, not independent evidence.
- The overall score is exactly 65.5; exact .5 rounds up (Decision 0010).
- Citation check incomplete — recheck later (0058: three candidates read in full, one counted; Semantic Scholar still unavailable for SAFE).
Sources
- NeurIPS 2024 paper (authors' claims): 72.0% agreement with crowdsourced labels on 16,011 facts — vendor or maintainer, accessed 2026-10-16
- GitHub repository: code, README and licences (authors) — vendor or maintainer, accessed 2026-10-16
- Zhao et al. (Findings of ACL 2025): 84.48% agreement on 786 biography facts; authors propose an alternative method — independent performance, accessed 2026-10-16
- FactBench (ACL 2025): response-level Pearson 0.95 (factual labels) / 0.60 (other labels); authors propose an alternative method — independent performance, accessed 2026-10-16
- VeriScore (Findings of EMNLP 2024): critique of SAFE's claim extraction; authors propose an alternative method — independent performance, accessed 2026-10-16
- VeriFact (EMNLP 2025; University of Michigan): on 1,168 human-annotated facts, 56.7% of SAFE-extracted facts were incomplete (VeriFact 22.5%) and SAFE covered 77.5% of human-identified facts; the annotated set was chosen for responses with many extraction problems; authors propose an alternative method — independent performance, accessed 2026-10-17
- VentureBeat: reports Gary Marcus's critique of the "superhuman" framing (journalism) — independent other, accessed 2026-10-16; could not be fetched automatically (HTTP 429); read via search index
- ForkLog: quotes the same Gary Marcus critique (journalism) — independent other, accessed 2026-10-16