OpenFactCheck (open-source)
- Category
- Verification
- Type
- Open-source factuality framework (PyPI openfactcheck; repository licence AGPL-3.0, PyPI metadata says GPL-3.0), MBZUAI researchers; GitHub openfactcheck-research/openfactcheck
- Scope
- The open-source OpenFactCheck library (PyPI 0.0.17 July 2024 up to 1.1.2; the 2.0 alpha series, 2.0.0a1 to 2.0.0a10 in 2026, is not assessed): CustChecker/ResponseEval (configurable fact-checker built from FacTool, Factcheck-GPT and other modules), LLMEval and CheckerEval. Run with user-chosen models and retrievers. The hosted web app is not covered.
- Source
- GitHub repository
- Last reviewed
- 2026-10-19 (what this means)
Summary
OpenFactCheck is an open-source framework from MBZUAI researchers for building configurable fact-checkers, measuring how factual language models are, and benchmarking fact-checkers. Its fact-checking mostly reuses parts of earlier systems (FacTool, Factcheck-GPT). No independent evaluation that ran it was found; the only figures come from its developers' EMNLP 2024 demo and COLING 2025 papers. It is actively maintained (2.0 alpha releases in 2026). No CVEs or security advisories were found. Insufficient evidence for an overall score. Informational only; not a guarantee, endorsement, or certification.
Status: Insufficient evidence — no overall score is given (fewer than three independent performance sources). Independent performance sources: 0.
Factor notes (no scores)
| Factor | Weight | Score | Note |
|---|---|---|---|
| Completion rate | 30% | — | No independent results. Developer papers benchmark the component checkers (e.g. Factcheck-GPT with GPT-4 reaching false-claim F1 0.63 on Factcheck-Bench) rather than a single OpenFactCheck configuration. Not scored. |
| Disputes | 20% | — | None found: no CVEs, no GitHub security advisories, no legal or regulatory action. Licence metadata differs between the repository (AGPL-3.0) and PyPI (GPL-3.0). Not scored. |
| Adherence to limits | 25% | — | No independent tests found. Not scored. |
| Evidence quality | 25% | — | Zero independent performance sources; below the three-source minimum. Not scored. |
See the methodology for the insufficient-evidence rule.
Scope and version note
This entry covers the open-source OpenFactCheck library (PyPI openfactcheck) up to 1.1.2, run with models and retrievers chosen by the user. Its fact-checking largely reuses components from earlier systems (FacTool, Factcheck-GPT), so results depend on the configuration. The 0.x to 1.1.2 releases are treated as continuous; the 2.0 alpha series (2026) has not been assessed. The hosted web app is not covered; it did not respond on 17 October 2026.
Licence note
The repository states AGPL-3.0; the PyPI package metadata states GPL-3.0.
Related entry and shared authors
Loki and OpenFactCheck share MBZUAI-linked authors (Yuxia Wang, Minghan Wang, Preslav Nakov); Libr-AI is MBZUAI-linked. They are separate code bases with different purposes and are assessed separately. See Loki (open-source).
Security
No CVEs or GitHub security advisories were found (checked 17 October 2026).
Uncertainties
- Semantic Scholar citation check (17 Oct 2026) covered all papers citing the three OpenFactCheck papers; none ran OpenFactCheck as a tested system. An OpenAlex full-text recheck (Decision 0080, 19 Oct 2026) read 13 newer citing papers in full; none ran it either. Papers without open full text were checked by abstract only.
- OpenFactCheck is a framework that combines other checkers; "performance" depends on the configuration chosen.
- The 2.0 alpha series may be a substantial rewrite; continuity is unassessed beyond release history.
Sources
Other sources (not counted)
- UFO (AAAI 2026): describes OpenFactCheck and Loki as tied to a single or fused fact source with limited flexibility; does not run either. Not counted as performance — other, accessed 2026-10-17
- Independent written analysis (May 2026) based on the published papers; not a hands-on test. Not counted — other, accessed 2026-10-17
Developer and maintainer sources (not counted)
- OpenFactCheck paper (COLING 2025; Wang, Wang, Iqbal, Georgiev, Geng, Gurevych, Nakov): FactBench of 4,507 claims; checkers detect true claims more reliably than false ones. Developer results, not counted — vendor or maintainer, accessed 2026-10-17
- OpenFactCheck demo paper (EMNLP 2024 System Demonstrations). Developer, not counted — vendor or maintainer, accessed 2026-10-17
- Repository (moved from yuxiaw/openfactcheck and mbzuai-nlp/openfactcheck): AGPL-3.0, last push 15 Aug 2026; PyPI 49 releases, latest 2.0.0a10 (13 Jul 2026); no security advisories — vendor or maintainer, accessed 2026-10-17