Julius AI

Summary

Julius AI is a commercial AI data-analysis tool that generates and runs code from natural-language prompts. Four limited independent evaluations were found. A 2026 peer-reviewed study found high accuracy on text-based statistics questions (86.49% on a statistical-literacy test; 78.6% on statistical data questions) and lower accuracy on causal questions and items needing figures. Two education studies from one research group found errors on a PDF math worksheet and a multiplication prompt, and a peer-reviewed comparison tested Julius with code execution disabled. Informational only; not a guarantee, endorsement, or certification.

Overall score: 59 · Evidence grade: B- · Independent sources: 4 performance (2 peer-reviewed), 7 other

Evidence grade describes the amount and type of evidence, not the agent's quality. Grade B-: 4 counted sources (3–4 band; see methodology). Three of the four performance sources are limited (code execution disabled; a single prompt; a PDF worksheet in a venue whose review could not be confirmed).

Factor breakdown

0.30 × 55 + 0.20 × 80 + 0.25 × 50 + 0.25 × 55 = 58.75, rounded to 59 (rounding rule).

Disclosures for counted sources

• Quality & Quantity (2026): the authors state that the vendor provided complimentary access to the tool during the initial stages of the work. No vendor authors; no conflicts declared.
• IJRSI (2026) and the Ohio Journal of School Mathematics study (2025) share three authors (Sawyer, Aga, Sutherland). They are counted as separate studies because they use different tasks, dates and tool sets.

Uncertainties

• Only one counted study (Quality & Quantity 2026) tests Julius on data-analysis tasks as normally used; it does not state the version, underlying model or test date, and the vendor gave the authors complimentary access.
• The IJRSI study is in a venue whose review could not be confirmed (accepted 5 days after receipt), tests a PDF mathematics worksheet rather than a data file, and shares three authors with the Ohio Journal of School Mathematics study.
• The Applied Sciences comparison contains internal inconsistencies in its stated test dates and software versions.
• Benchmarks published by competing products are not independent.
• Privacy, retention, training-use, and compliance statements are vendor statements and were not independently verified.
• The product and its billing model changed in 2026; no counted source tests the 2026 product, and older evidence may not reflect it.
• A 2025 Teaching Statistics illustration (Vilakati) is listed but not counted; the line between a documented hands-on test and an illustration is a judgment call.

Sources

• Abbasi et al. (2026), Applied Sciences 16(2):669: comparison on bibliometric tasks (peer-reviewed; limited: Julius code execution disabled) — independent performance, accessed 2026-10-16; could not be fetched automatically (HTTP 403)
• Ohio Journal of School Mathematics (2025): generative AI chatbots and multiplication (single prompt; education journal) — independent performance, accessed 2026-10-16
• Marmolejo-Ramos, Rojas-Galeano, Ciria and Tejada (2026), Quality & Quantity (Flinders University, Universidad Distrital, UNAM, Universidade Federal de Sergipe; peer-reviewed, open access): Julius on BLIS (32/37, 86.49%; all errors on items with a figure), QRDATA (statistical 78.6%, social science 57.1%, causal 54.5%) and MathBench probability and statistics subdomains (43.7-100%), plus a modelling demonstration. Disclosure: the vendor provided complimentary access to the tool during the initial stages of the work — independent performance, accessed 2026-10-19
• Wolfe, Sawyer, Aga, Park, Sutherland and Smiley (2026), IJRSI 13(9):1506-1530 (James Madison University, Drake University): eleven generative AI tools on a nine-question NAEP 2013 grade-12 mathematics PDF, one prompt; Julius identified 4 of 9 questions, 5 omissions, 0 of 5 multiple-choice correct. Limited: a PDF worksheet rather than a data file; journal states peer review, but acceptance came 5 days after receipt, so not treated as peer-reviewed. Same research group as the Ohio Journal of School Mathematics study (a different study) — independent performance (limited), accessed 2026-10-19
• Discover Education (2024): tutorial on generative AI in mixed-methods analysis (tutorial; not a correctness evaluation) — independent other, accessed 2026-10-16
• Veredas do Direito (2026): Julius in postgraduate statistical analysis (student-learning study) — independent other, accessed 2026-10-16; bibliographic record (OpenAlex). The publisher page for DOI 10.18623/rvd.v23.7013 was unavailable as of 2026-10-17 (HTTP 404)
• Vilakati (2025), Teaching Statistics (University of the Free State; peer-reviewed): illustration on a hypothetical ten-row dataset; with a plain prompt, Julius (GPT-4o; similar with Claude 3.7 Sonnet) ran a t-test without checking its assumptions, also after outliers were added; with more context and Claude 3.7 Sonnet the result was correct. An illustration on constructed data rather than a measured accuracy test, so not counted (as for the Chang article in Decision 0058) — independent other, accessed 2026-10-19; journal page could not be fetched automatically; preprint read in full (https://doi.org/10.20944/preprints202506.2075.v1)
• CASRAI guide: Julius AI for research data analysis (guidance; not a test) — independent other, accessed 2026-10-16
• University of South Florida REU poster: AI in ocean traffic monitoring (student poster) — independent other, accessed 2026-10-16; could not be fetched automatically (HTTP 403)
• Rows: AI Spreadsheet Benchmark (competitor benchmark; not independent) — competitor, accessed 2026-10-16
• Basedash: BI Bench results (competitor benchmark; not independent) — competitor, accessed 2026-10-16
• Fastero: ChatGPT vs Julius AI comparison (July 2026, updated September 2026; five runs of one churn analysis; the publisher sells an analytics platform and recommends it; competitor comparison; not independent) — competitor, accessed 2026-10-19
• DA-bench listing for Julius (benchmark listing; operator not identified; no score shown) — independent other, accessed 2026-10-16
• Newsletter test of AI quantitative analysis tools (anecdotal; newsletter) — anecdotal, accessed 2026-10-16
• Julius docs: privacy and data security (vendor statement) — vendor or maintainer, accessed 2026-10-16
• Julius security page (vendor-stated compliance claims) (vendor statement) — vendor or maintainer, accessed 2026-10-16
• Julius privacy policy (vendor statement) — vendor or maintainer, accessed 2026-10-16
• Julius docs: file retention (vendor statement) — vendor or maintainer, accessed 2026-10-16
• Julius docs: usage credits by plan (vendor statement) — vendor or maintainer, accessed 2026-10-16

Decision records: 0007; 0058 (follow-up review); 0076 (evidence pass 2).