Use Decisions API grounded web search to fact check claims

Ask a decision model whether Navi Pillay was awarded the 2026 Nobel Peace Prize and it returns 0.01. She was. Without a search the model answers from the state alone, and the state says nothing about last month. With mode: "web_grounding" the same request returned 1.00. Put 600 characters of unrelated text in front of the claim and the grounded request returned 0.07, with five office-move pages in search_results: the query is built from the start of the state, and the claim was not in it. This recipe shows how to shape the request so the search sees your claim, how to confirm it did, and how to measure what grounding changes. A decision model returns a probability, not prose. Without mode, it answers from the state you send alone. With mode: "web_grounding", the API builds a search query from the question text and the start of your state, runs a Fast Search, appends the results to the state as evidence, and the model answers from that. The response adds search_results, the evidence the model saw. Your code compares the probability to two cutoffs and marks the claim supported, contradicted, or review. The recipe runs 24 labeled claims, 8 about facts settled for years and 16 about events from September and October 2026; without grounding no true recent claim printed above 0.24, and with it 24 of 24 came back right.

Why web grounding

One call instead of two APIs stitched together. One POST /v1/decisions replaces a Search API request, prompt assembly, and a second request.
A number you can threshold. p >= 0.80 is supported, p <= 0.20 is contradicted, and the middle goes to a person. The cutoffs are constants in your code, not instructions to the model.
The search sees the start of your state. The query is the question text and then as much of your state as fits in 256 bytes, so where you put the claim decides what the search looks for, and search_results shows you what it found.

What you need

Python 3.10 or newer on macOS or Linux. The shell commands use Unix syntax.
A Perplexity API key. Every key can call the Decisions API. A grounded request bills input tokens as usual plus $1.00 per 1,000 searches.

Set up

Everything lives in one directory. You create the first six files from this page; the script writes out/ on every run. The script posts to https://api.perplexity.ai/v1/decisions with httpx, like the ticket triage recipe, so the exact body the gateway reads is on the page. With the Perplexity Python library (0.46.1), which has no named mode parameter or search_results field yet, pass extra_body={"mode": "web_grounding"} to client.decisions.create and read the results from response.to_dict()["search_results"].

decisions-api-web-grounding/
├── requirements.txt       # two pinned dependencies
├── claims.jsonl           # 24 labeled claims, one JSON object per line
├── query.py               # the gateway's search query rule, mirrored
├── factcheck.py           # check and eval
├── test_query.py          # the gateway's own test cases, offline
├── test_factcheck.py      # offline tests, no network
├── out/                   # one JSON file per run, written by the script
└── .venv/                 # virtual environment from the install step

requirements.txt

httpx==0.28.1
pytest==9.1.1

Create the directory, save requirements.txt in it, then create a virtual environment, install, and export your key.

Install and set the key

mkdir decisions-api-web-grounding && cd decisions-api-web-grounding
# save requirements.txt here, then continue
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
export PERPLEXITY_API_KEY="your-api-key-here"

The claims

Each line has an id, the claim, a label (true or false, checked against the linked source), a kind, and the source URL. stable claims are facts that have not changed in years. recent claims are about events from September and October 2026: Nobel Prizes, the World Series, the Ballon d’Or, a Python release, a Federal Reserve meeting. 13 are true and 11 are false. The script sends only the claim text; the label and the source stay on your side. The longest claim is 100 bytes, so with the 18-byte question every claim fits the query in full when it comes first.

claims.jsonl

{"id": "s01", "claim": "Python 3.12 was released in October 2023.", "label": true, "kind": "stable", "source": "https://www.python.org/downloads/release/python-3120/"}
{"id": "s02", "claim": "Rust 1.0 was released in May 2015.", "label": true, "kind": "stable", "source": "https://blog.rust-lang.org/2015/05/15/Rust-1.0.html"}
{"id": "s03", "claim": "The Great Barrier Reef lies off the coast of Queensland, Australia.", "label": true, "kind": "stable", "source": "https://en.wikipedia.org/wiki/Great_Barrier_Reef"}
{"id": "s04", "claim": "HTTP status code 404 means the requested resource was not found.", "label": true, "kind": "stable", "source": "https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404"}
{"id": "s05", "claim": "The Linux kernel was first released in 1998.", "label": false, "kind": "stable", "source": "https://en.wikipedia.org/wiki/Linux_kernel"}
{"id": "s06", "claim": "Mount Kilimanjaro is the highest mountain in South America.", "label": false, "kind": "stable", "source": "https://en.wikipedia.org/wiki/Mount_Kilimanjaro"}
{"id": "s07", "claim": "Git was created by Guido van Rossum.", "label": false, "kind": "stable", "source": "https://en.wikipedia.org/wiki/Git"}
{"id": "s08", "claim": "PostgreSQL is licensed under the GNU General Public License.", "label": false, "kind": "stable", "source": "https://www.postgresql.org/about/licence/"}
{"id": "r01", "claim": "Navi Pillay was awarded the 2026 Nobel Peace Prize.", "label": true, "kind": "recent", "source": "https://www.nobelprize.org/prizes/peace/2026/summary/"}
{"id": "r02", "claim": "Francis Halzen won the 2026 Nobel Prize in Physics for the IceCube Neutrino Observatory.", "label": true, "kind": "recent", "source": "https://www.nobelprize.org/prizes/physics/2026/summary/"}
{"id": "r03", "claim": "Canadian author Anne Carson won the 2026 Nobel Prize in Literature.", "label": true, "kind": "recent", "source": "https://www.nobelprize.org/prizes/literature/2026/press-release/"}
{"id": "r04", "claim": "Henri Kagan and Kenso Soai shared the 2026 Nobel Prize in Chemistry.", "label": true, "kind": "recent", "source": "https://www.nobelprize.org/prizes/chemistry/2026/summary/"}
{"id": "r05", "claim": "The Milwaukee Brewers won the 2026 World Series, the first title in franchise history.", "label": true, "kind": "recent", "source": "https://www.mlb.com/news/brewers-win-world-series-2026"}
{"id": "r06", "claim": "Python 3.15.0 was released on October 9, 2026.", "label": true, "kind": "recent", "source": "https://docs.python.org/3/whatsnew/3.15.html"}
{"id": "r07", "claim": "Harry Kane won the 2026 men's Ballon d'Or.", "label": true, "kind": "recent", "source": "https://www.espn.com/soccer/story/_/id/50236983/harry-kane-2026-ballon-dor-kylian-mbappe-lamine-yamal"}
{"id": "r08", "claim": "The Federal Reserve held its target range at 3.75 to 4 percent at its October 28, 2026 meeting.", "label": true, "kind": "recent", "source": "https://www.federalreserve.gov/newsevents/pressreleases/monetary20261028a.htm"}
{"id": "r09", "claim": "Haruki Murakami won the 2026 Nobel Prize in Literature.", "label": false, "kind": "recent", "source": "https://www.nobelprize.org/prizes/literature/2026/press-release/"}
{"id": "r10", "claim": "The Tampa Bay Rays won the 2026 World Series.", "label": false, "kind": "recent", "source": "https://www.mlb.com/news/brewers-win-world-series-2026"}
{"id": "r11", "claim": "Lamine Yamal won the 2026 men's Ballon d'Or.", "label": false, "kind": "recent", "source": "https://www.espn.com/soccer/story/_/id/50236983/harry-kane-2026-ballon-dor-kylian-mbappe-lamine-yamal"}
{"id": "r12", "claim": "The Federal Reserve cut its target rate by a quarter point at its October 28, 2026 meeting.", "label": false, "kind": "recent", "source": "https://www.federalreserve.gov/newsevents/pressreleases/monetary20261028a.htm"}
{"id": "r13", "claim": "Apple unveiled the iPhone 18 Pro at an event in October 2026.", "label": false, "kind": "recent", "source": "https://en.wikipedia.org/wiki/IPhone_18_Pro"}
{"id": "r14", "claim": "The 2026 Nobel Prize in Physics was awarded for the discovery of room-temperature superconductivity.", "label": false, "kind": "recent", "source": "https://www.nobelprize.org/prizes/physics/2026/summary/"}
{"id": "r15", "claim": "Daron Acemoglu and Simon Johnson won the 2026 Nobel Prize in Economic Sciences.", "label": false, "kind": "recent", "source": "https://www.reuters.com/world/hall-woodford-win-2026-nobel-economics-prize-2026-10-12/"}
{"id": "r16", "claim": "Karl Deisseroth, Peter Hegemann and Georg Nagel won the 2026 Nobel Prize in Physiology or Medicine.", "label": true, "kind": "recent", "source": "https://www.nobelprize.org/prizes/medicine/2026/summary/"}

Shape the request

Before the model sees anything, the gateway builds one search query per question. The query is the question’s instructions (and its criteria text, if you wrote any), then the text of your state, in that order, cut to 256 bytes of normalized UTF-8. When the state has text, the question part gets at most 128 bytes; the separator takes one more, so a question that uses all 128 leaves 127 for the state. The state part is a prefix: the first bytes of your state, as many as fit. A word the limit would cut is dropped unless at least 32 bytes of it fit, so an unspaced script keeps its start, and once something is cut nothing later fills the gap. Runs of whitespace and control characters collapse to one space, the ends are trimmed, a code point is never split, and data: URLs are skipped. The search runs with that query, the gateway appends the results to a string state after an Evidence: line, the model answers, and the response carries the same results in search_results: title, URL, snippet, and dates, or [] when nothing was added. So keep the question short, because every byte of question text is a byte the claim cannot use: “Is the claim true?” is 18 bytes, which leaves 237 for the claim. And send the claim alone, or keep it at the start, before any other context. Text in front of the claim uses up those bytes: once it is long enough, the limit cuts the claim mid-sentence and then drops it altogether, and the search runs on the text in front instead. What the search sees shows what that did here. query.py is a port of the gateway’s rule for the shape this recipe sends: a string state and one noul question with plain instructions. search_query gives you the query text, and claim_in_query tells you before sending whether the complete claim text is inside it; a claim the limit cuts in the middle counts as outside. The script runs that check on every claim, prints the count, and warns when any claim is outside. For criteria, choice and score questions, and object or array state, the helper raises instead of guessing; Web grounding has the rule for those.

The query rule

normalize_prefix is the normalization above, applied to one part with a byte budget; the second value it returns says whether the input was truncated. Text accumulates parts up to a budget and, after a cut, pins its budget at the current length so a later part cannot fill a dropped word’s gap. search_query runs the state through a 256-byte Text, the question text through a 128-byte one when the state has text, and joins the two in a final 256-byte Text. It counts UTF-8 bytes, so a claim with accented characters fits fewer characters than one in plain ASCII. is_data_url trims the same characters Go’s TrimSpace does, which is narrower than Python’s strip. claim_in_query normalizes the claim the same way and looks for the whole of it in the query.

query.py

"""The search query the gateway builds for pplx-decider web grounding, mirrored for the shapes this recipe
sends. The rule: one query per question, the question text first and then the start of the `state`, 256
bytes in all, whitespace-normalized and cut at word boundaries; with `state` text the question part gets
at most 128 bytes. The rule is documented under Web grounding in the Decisions quickstart
(docs/decisions/quickstart#web-grounding)."""

import unicodedata

MAX_QUERY_BYTES = 256  # the Search API per-query limit: question text, then the start of the state
MAX_DROPPED_CUT = 32  # a word cut by the limit is dropped when fewer bytes than this fit


def normalize_prefix(s: str, n: int) -> tuple[str, bool]:
    """The gateway's normalization: whitespace and control runs become one space, ends trimmed, at most n
    bytes without splitting a code point. A word cut by the limit is dropped when fewer than MAX_DROPPED_CUT
    bytes of it fit. The flag reports that s did not fit."""
    out, pending, cut, truncated, last_word = bytearray(), False, False, False, 0
    for char in s:
        sep = char.isspace() or unicodedata.category(char) == "Cc"
        if len(out) >= n or (pending and out and len(out) + 2 > n):
            cut, truncated = not sep and not pending, True
            break
        if sep:
            pending = True
            continue
        need = len(char.encode("utf-8")) + (1 if pending and out else 0)
        if len(out) + need > n:
            cut, truncated = not pending, True
            break
        if pending and out:
            last_word = len(out)
            out += b" "
        pending = False
        out += char.encode("utf-8")
    written = len(out) - last_word - (1 if last_word else 0)
    if cut and written < MAX_DROPPED_CUT:
        return out[:last_word].decode("utf-8"), True
    return out.decode("utf-8"), truncated


class Text:
    """Normalized words up to max bytes. After a cut, later parts cannot fill the gap: it stays a prefix."""

    def __init__(self, max_bytes: int) -> None:
        self.s, self.max = "", max_bytes

    def room(self) -> int:  # bytes another word may take, after its separator
        used = len(self.s.encode("utf-8"))
        return self.max - used - (1 if used else 0)

    def add(self, s: str) -> None:
        if self.room() <= 0:
            return
        part, truncated = normalize_prefix(s, self.room())
        if part:
            self.s = f"{self.s} {part}" if self.s else part
        if truncated:
            self.max = len(self.s.encode("utf-8"))


GO_SPACE = ("\t\n\v\f\r \x85\xa0\u1680\u2000\u2001\u2002\u2003\u2004\u2005\u2006\u2007\u2008\u2009\u200a"
            "\u2028\u2029\u202f\u205f\u3000")  # Go's unicode.IsSpace, what strings.TrimSpace removes


def is_data_url(s: str) -> bool:
    """The gateway's check: after Go's TrimSpace set (not Python's wider strip), the text starts with data:."""
    return s.strip(GO_SPACE)[:5].lower() == "data:"


def search_query(state: str, question: dict) -> str:
    """The search text the gateway builds: the question text, then the start of the state, MAX_QUERY_BYTES in
    all. With state text the question gets at most half. Ports the gateway for a string state and one
    `noul` question with string instructions and no criteria; other shapes raise instead of guessing."""
    text = question.get("instructions")
    if (question.get("type") != "noul" or "criteria" in question or not isinstance(text, (str, type(None)))
            or not isinstance(state, str)):
        raise ValueError("search_query covers a string state and a noul question with string instructions only")
    state_part = Text(MAX_QUERY_BYTES)
    if not is_data_url(state):
        state_part.add(state)
    question_part = Text(MAX_QUERY_BYTES // 2 if state_part.s else MAX_QUERY_BYTES)
    if text and not is_data_url(text):
        question_part.add(text)
    query = Text(MAX_QUERY_BYTES)
    query.add(question_part.s)
    query.add(state_part.s)
    return query.s


def claim_in_query(claim: str, state: str, question: dict) -> bool:
    """True when the complete claim text, normalized the same way, appears in the search query built from state.
    A claim the limit cuts in the middle counts as outside, even though its start is in the query."""
    return normalize_prefix(claim, len(claim.encode("utf-8")))[0] in search_query(state, question)

The tests for it are the gateway’s own cases, ported: every normalizePrefix assertion, the prefix-after-a-cut case, the half and whole budgets, the start of a long state, an unspaced Japanese state, data: URLs in either position and behind a control character, the shapes the port refuses, and where a padded claim stops being inside the query, including the padding that keeps the claim’s start and cuts its end.

test_query.py

"""Offline checks for query.py, ported from the gateway's own test cases. Run: python -m pytest -q"""

import pytest

from factcheck import QUESTION, pad
from query import MAX_DROPPED_CUT, MAX_QUERY_BYTES, Text, claim_in_query, normalize_prefix, search_query

LONG = "word " * 200
WORDS = {"type": "noul", "instructions": LONG}


@pytest.mark.parametrize(("s", "n", "expected", "truncated"), [
    ("  a \t\n b  ", 100, "a b", False),
    ("a  b", 2, "a", True),  # a separator that would overflow stops the output
    ("é é", 3, "é", True),  # never split a code point
    ("   ", 10, "", False),
    ("a" + " " * (1 << 20) + "b", 2, "a", True),
    (" b", 1, "b", False),
    ("a\x00\x1b[0m\x7fb", 100, "a [0m b", False),  # control characters separate words
    ("alpha shipping", 10, "alpha", True),  # a word cut by the limit is dropped
    ("ab cd", 4, "ab", True),  # a word that reaches the limit mid-word is dropped
    ("ab cd e", 5, "ab cd", True),  # a complete last word is kept
    ("shipping news", 4, "", True),  # a short fragment is dropped even when it is the only word
    ("a b", 3, "a b", False),
    ("a " + "x" * 40, MAX_DROPPED_CUT + 2, "a " + "x" * MAX_DROPPED_CUT, True),  # a long cut word keeps a prefix
])
def test_normalize_prefix_matches_the_gateway(s: str, n: int, expected: str, truncated: bool) -> None:
    assert normalize_prefix(s, n) == (expected, truncated)


def test_text_stays_a_prefix_after_a_cut() -> None:
    t = Text(12)
    t.add("Order shipping")
    t.add("late")
    assert t.s == "Order"  # a later part must not fill the dropped word's gap


def test_query_is_capped_at_the_search_limit() -> None:
    query = search_query(LONG, WORDS)
    assert len(query.encode()) <= MAX_QUERY_BYTES and query.startswith("word word") and not query.endswith(" ")
    half, whole = MAX_QUERY_BYTES // 2 // len("word "), MAX_QUERY_BYTES // len("word ")
    assert search_query("Order", WORDS) == ("word " * half).strip() + " Order"  # the question keeps half
    assert search_query("", WORDS) == ("word " * whole).strip()  # no state: the whole budget
    microsoft = search_query("Microsoft fiscal 2025", WORDS)
    assert microsoft != search_query("Apple fiscal 2025", WORDS) and microsoft.endswith(" Microsoft fiscal 2025")


def test_query_searches_the_start_of_a_long_state() -> None:
    text = "Did Microsoft's fiscal 2023 revenue exceed $240 billion?"
    trajectory = "step: the agent read a page and wrote notes. " * 50
    query = search_query(trajectory, {"type": "noul", "instructions": text})
    assert query.startswith(text + " step: the agent read a page and wrote notes.") and len(query.encode()) <= 256
    state_only = search_query(trajectory, {"type": "noul"})  # no question text: the state
    assert state_only == trajectory.strip()[:len(state_only)] and 200 < len(state_only.encode()) <= 256


def test_query_keeps_an_unspaced_state_prefix() -> None:
    state = "注文番号12は配送が遅れました" * 20
    query = search_query(state, {"type": "noul", "instructions": "Was the order late?"})
    assert len(query.encode()) <= 256 and query.startswith("Was the order late? " + state[:14])


@pytest.mark.parametrize(("state", "question", "expected"), [
    ("  Order   #12\nshipped late ", {"type": "noul", "instructions": "Was it late?"},
     "Was it late? Order #12 shipped late"),
    ("data:text/plain,claim", QUESTION, QUESTION["instructions"]),  # a data: state is skipped
    ("claim", {"type": "noul", "instructions": " DATA:text/plain,q"}, "claim"),  # so is a data: question
    ("claim", {"type": "noul", "instructions": None}, "claim"),
    ("\x1cdata:text/plain,claim", QUESTION, "Is the claim true? data:text/plain,claim"),  # \x1c is not TrimSpace
    ("\u3000data:text/plain,claim", QUESTION, QUESTION["instructions"]),  # ideographic space is
])
def test_search_query_normalizes_and_skips_data_urls(state: str, question: dict, expected: str) -> None:
    assert search_query(state, question) == expected


@pytest.mark.parametrize(("state", "question"), [
    ("x", {"type": "noul", "instructions": "Is it true?", "criteria": {"true": "yes", "false": "no"}}),
    ("x", {"type": "noul", "instructions": {"ask": "Is it true?"}}),
    ("x", {"type": "choice", "instructions": "Pick one.", "options": ["a", "b"]}),
    ("x", {"type": "score", "instructions": "Rate it."}),
    ("x", {"instructions": "Is it true?"}),  # no type
    ("x", {"type": "NOUL", "instructions": "Is it true?"}),  # the gateway matches the type exactly
    (["x"], QUESTION)])
def test_search_query_refuses_shapes_it_does_not_port(state, question: dict) -> None:
    with pytest.raises(ValueError):
        search_query(state, question)


@pytest.mark.parametrize(("state", "inside"), [
    ("Python 3.12 was released in October 2023.", True),
    ("Python  3.12 was released\nin October 2023.", True),  # the claim is normalized the same way
    (pad("Python 3.12 was released in October 2023.", 0, 600), True),  # claim first: filler follows it
    (pad("Python 3.12 was released in October 2023.", 150, 0), True),  # 18 + 1 + 150 + 1 + 41 bytes fit
    (pad("Python 3.12 was released in October 2023.", 200, 0), False),  # the limit cuts the claim's last word
    (pad("Python 3.12 was released in October 2023.", 600, 0), False),  # the claim starts past the limit
])
def test_claim_in_query_depends_on_where_the_claim_sits(state: str, inside: bool) -> None:
    assert claim_in_query("Python 3.12 was released in October 2023.", state, QUESTION) is inside


def test_a_cut_claim_keeps_only_its_start() -> None:
    query = search_query(pad("Python 3.12 was released in October 2023.", 200, 0), QUESTION)
    assert query.endswith(" Python 3.12 was released in October") and len(query.encode()) <= 256
    assert search_query(pad("Python 3.12 was released in October 2023.", 600, 0), QUESTION).find("Python") == -1

Constants and the padding helper

QUESTION is the one question every request asks. The cutoffs and the three acceptance bounds are the policy; the model never sees them. PRICE_PER_M_INPUT and PRICE_PER_SEARCH are the published rates, used to turn usage.input_tokens and the request count into a cost per 1,000 claims. FILLER is the office-move text the padding experiments put around each claim; it is in the file so the experiments are reproducible. pad builds the experiment input: the claim on its own line after before characters of FILLER and before after more.

Constants and pad()

factcheck.py (part 1 of 4)

"""Fact-check claims against the web with the Decisions API. One request per claim: the claim is the
`state`, one `noul` question asks whether it is true, and `mode: "web_grounding"` adds a web search."""

import argparse
import json
import os
import sys
import time
from dataclasses import asdict, dataclass, field
from pathlib import Path

import httpx

from query import claim_in_query

DECISIONS_URL = "https://api.perplexity.ai/v1/decisions"
MODEL = "pplx-decider-latest"  # selects pplx-decider-v1.2
QUESTION_NAME = "is_true"
QUESTION = {"type": "noul", "instructions": "Is the claim true?"}

SUPPORTED_MIN = 0.80  # p >= this: supported. Your code applies the cutoffs, not the model.
CONTRADICTED_MAX = 0.20  # p <= this: contradicted; anything between: review
MAX_MISSED_FAILURES = 0  # acceptance bounds the eval reports against: false claims marked supported
MAX_FALSE_CONTRADICTED_RATE = 0.05  # true claims marked contradicted, as a share of claims
MAX_REVIEW_RATE = 0.25  # share of claims sent to a person

PRICE_PER_M_INPUT = {"pplx-decider-latest": 0.02, "pplx-decider-v1.2": 0.02,  # $ per million input tokens
                     "pplx-decider-flash-latest": 0.01, "pplx-decider-flash-v1.2": 0.01}
PRICE_PER_SEARCH = 0.001  # dollars per web search when grounded
RETRIES = 3  # on 429 and 5xx; waits Retry-After seconds when it is an integer, else 1, 2, 4
MAX_WAIT_S = 30  # cap on any single wait
OUT_DIR = Path("out")

FILLER = (  # neutral text for --pad-before and --pad-after; nothing in it relates to any claim
    "The facilities team confirmed the office move for the second week of the quarter. "
    "Badge readers on the third floor will be replaced on Monday, and visitor parking "
    "moves to the lower level. The updated seating chart goes out by email once the "
    "furniture vendor confirms delivery. Please label boxes with your team name. ")


def pad(claim: str, before: int, after: int) -> str:
    """The claim on its own line, after `before` characters of FILLER and before `after` more."""
    def filler(n: int) -> list[str]:
        return [(FILLER * (n // len(FILLER) + 1))[:n]] if n else []
    return "\n".join(filler(before) + [claim] + filler(after))

One request per claim

decide sends the claim as state with the one question, and adds "mode": "web_grounding" only when asked. 429 and 5xx responses are retried up to three times, waiting Retry-After seconds when the header is a whole number and 1, 2, then 4 seconds when it is absent or anything else, never more than 30 seconds. Any other error raises DecisionsError with the status and error.message from the body, and so does a 200 whose body is not the expected shape, including a probability outside [0, 1], a negative token count, or a search result without a url. The Decision it returns holds the probability from answers.is_true.noul, usage.input_tokens, the wall time for the claim with retries included, the number of POSTs it took, and the search_results URLs, [] when the key is absent. The POST count matters when grounded: a search is billed when it returns, even if the decision then fails and you retry, so every grounded attempt may be a billed search. mode accepts one value. Anything else returns 400:

{"error":{"code":null,"message":"mode must be \"web_grounding\" when provided","param":"mode","type":"invalid_request_error"}}

decide()

factcheck.py (part 2 of 4)

class DecisionsError(Exception):
    """A Decisions API request that did not return an answer."""


@dataclass
class Decision:
    probability: float
    input_tokens: int
    latency_s: float  # wall time for the claim, retries included
    attempts: int  # POSTs sent; when grounded, each one may have run a billed search
    results: list[str] = field(default_factory=list)  # search_results URLs; [] without mode or evidence


def is_probability(p: object) -> bool:
    return isinstance(p, (int, float)) and not isinstance(p, bool) and 0.0 <= p <= 1.0


def decide(client: httpx.Client, claim: str, grounded: bool, model: str = MODEL) -> Decision:
    """POST one claim with one question. `mode` is sent only when grounded."""
    body: dict[str, object] = {"model": model, "state": claim, "questions": {QUESTION_NAME: QUESTION}}
    if grounded:
        body["mode"] = "web_grounding"
    started = time.perf_counter()
    for attempt in range(RETRIES + 1):
        response = client.post(DECISIONS_URL, json=body)
        if response.status_code == 200:
            try:
                data = response.json()
                p, tokens = data["answers"][QUESTION_NAME]["noul"], data["usage"]["input_tokens"]
                urls = [r["url"] for r in data.get("search_results") or []]
            except (ValueError, KeyError, TypeError) as error:
                raise DecisionsError(f"200 with an unexpected body: {error!r}") from None
            if not is_probability(p) or isinstance(tokens, bool) or not isinstance(tokens, int) or tokens < 0:
                raise DecisionsError(f"200 with an unexpected body: noul={p!r}, input_tokens={tokens!r}")
            return Decision(p, tokens, time.perf_counter() - started, attempt + 1, urls)
        if attempt == RETRIES or not (response.status_code == 429 or response.status_code >= 500):
            break
        retry_after = response.headers.get("Retry-After", "")
        time.sleep(min(int(retry_after) if retry_after.isdigit() else 2**attempt, MAX_WAIT_S))
    try:
        message = response.json()["error"]["message"]
    except (ValueError, KeyError, TypeError):
        message = response.text.strip()
    raise DecisionsError(f"{response.status_code}: {message}")

Verdicts and the scorecard

verdict is the whole policy: at or above 0.80 supported, at or below 0.20 contradicted, otherwise review. Both cutoffs are inclusive. evaluate scores one mode against the labels. A verdict is correct when supported meets a true label or contradicted meets a false one, wrong when they cross, and review is counted on its own. Wrong verdicts split into missed_failures, false claims marked supported, and false_contradictions, true claims marked contradicted. The Brier score is the mean squared gap between the probability and the label, so a confident wrong answer costs more than an unsure one. flips counts claims whose verdict changed between the two modes. grounded_attempts is the number of POSTs sent with mode set, an upper bound on billed searches; mean_search_results is the evidence count per claim; and cost per 1,000 claims is the mean input tokens at the model’s rate plus one search per attempt. It rejects empty input, mismatched lengths, labels that are not booleans, and probabilities outside [0, 1], in the baseline as well as the results. accepted holds a run to three bounds at once: MAX_MISSED_FAILURES = 0, MAX_FALSE_CONTRADICTED_RATE = 0.05, and MAX_REVIEW_RATE = 0.25. Each bound exists because a trivial policy passes the other two: marking everything contradicted has no missed failures and no reviews, and reviewing everything has no wrong verdicts at all. The tests check both of those fail and that the shape of the recorded grounded run passes. Set the bounds from your own tolerance, but keep all three.

verdict(), Summary, accepted(), and evaluate()

factcheck.py (part 3 of 4)

def verdict(p: float) -> str:
    return "supported" if p >= SUPPORTED_MIN else "contradicted" if p <= CONTRADICTED_MAX else "review"


@dataclass
class Summary:
    mode: str
    claims: int
    correct: int
    wrong: int
    review: int
    missed_failures: int  # false claims marked supported
    false_contradictions: int  # true claims marked contradicted
    flips: int | None  # None when no baseline was given
    brier: float
    mean_input_tokens: float
    mean_latency_s: float
    grounded_attempts: int  # upper bound on billed searches
    mean_search_results: float
    cost_per_1000: float


def accepted(s: Summary) -> bool:
    return (s.missed_failures <= MAX_MISSED_FAILURES and s.review <= MAX_REVIEW_RATE * s.claims
            and s.false_contradictions <= MAX_FALSE_CONTRADICTED_RATE * s.claims)


def evaluate(rows: list[dict], results: list[Decision], grounded: bool, model: str, baseline=None) -> Summary:
    """Score one mode against the labels. `baseline` is the other mode's results, for flips."""
    n = len(rows)
    for name, res in [(k, v) for k, v in (("results", results), ("baseline", baseline)) if v is not None]:
        if n == 0 or len(res) != n:
            raise ValueError(f"{name}: need one decision per row and at least one row")
        if not all(is_probability(d.probability) for d in res):
            raise ValueError(f"{name}: every probability must be a number in [0, 1]")
    if not all(isinstance(r["label"], bool) for r in rows):
        raise ValueError("every label must be true or false")
    labels = [r["label"] for r in rows]
    verdicts = [verdict(d.probability) for d in results]
    review = verdicts.count("review")
    correct = sum(v != "review" and (v == "supported") == label for v, label in zip(verdicts, labels))
    missed = sum(v == "supported" and not label for v, label in zip(verdicts, labels))
    false_contra = sum(v == "contradicted" and label for v, label in zip(verdicts, labels))
    brier = sum((d.probability - float(label)) ** 2 for d, label in zip(results, labels)) / n
    flips = None if baseline is None else sum(verdict(b.probability) != v for b, v in zip(baseline, verdicts))
    mean_tokens = sum(d.input_tokens for d in results) / n
    mean_latency = sum(d.latency_s for d in results) / n
    attempts = sum(d.attempts for d in results) if grounded else 0
    cost = 1000 * (mean_tokens * PRICE_PER_M_INPUT[model] / 1e6 + PRICE_PER_SEARCH * attempts / n)
    return Summary("grounded" if grounded else "ungrounded", n, correct, n - correct - review, review, missed,
                   false_contra, flips, brier, mean_tokens, mean_latency, attempts,
                   sum(len(d.results) for d in results) / n, cost)

Command line

load_claims reads the JSONL and names the line of anything malformed: a missing claim, a label that is not true or false, an empty file. check runs one mode over the file, eval runs both and reports flips. --pad-before N puts N characters of FILLER in front of each claim and --pad-after N puts them after it; both may be given. --model switches to the flash model, which bills input at half the rate. Before any request goes out, the script runs claim_in_query on every claim and warns when the complete text of any claim is outside its query. The table has one probability and one verdict per mode. check prints the search_results URLs under each row, so you can see the evidence behind a verdict; eval leaves them out of the table and adds mean search results to the grounded summary line. The summary is two lines per mode and ends with acceptance met or not met. Every run writes its rows, results, and summaries to out/ as JSON.

load_claims() and main()

factcheck.py (part 4 of 4)

def load_claims(path: str) -> list[dict]:
    """Read JSONL rows: string `id`, `claim`, `kind` and a bool `label`. Errors name the line."""
    rows = []
    for number, line in enumerate(Path(path).read_text(encoding="utf-8").splitlines(), 1):
        if not line.strip():
            continue
        try:
            row = json.loads(line)
        except ValueError as error:
            raise ValueError(f"{path}:{number}: not JSON ({error})") from None
        strings = isinstance(row, dict) and all(
            isinstance(row.get(k), str) and row[k].strip() for k in ("id", "claim", "kind"))
        if not strings or not isinstance(row.get("label"), bool):
            raise ValueError(f"{path}:{number}: needs string id, claim, kind and a true or false label")
        rows.append(row)
    if not rows:
        raise ValueError(f"{path}: no claims")
    return rows


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("command", choices=["check", "eval"])
    parser.add_argument("claims", help="JSONL with id, claim, label, kind per line")
    parser.add_argument("--grounded", action="store_true", help="check: send mode=web_grounding")
    parser.add_argument("--pad-before", type=int, default=0, help="put N chars of filler before each claim")
    parser.add_argument("--pad-after", type=int, default=0, help="put N chars of filler after each claim")
    parser.add_argument("--model", default=MODEL, choices=list(PRICE_PER_M_INPUT))
    args = parser.parse_args()
    if (args.command == "eval" and args.grounded) or min(args.pad_before, args.pad_after) < 0:
        parser.error("eval runs both modes (--grounded applies to check), and padding must be 0 or more")
    try:
        if not (api_key := os.environ.get("PERPLEXITY_API_KEY")):
            raise ValueError("set PERPLEXITY_API_KEY first")
        rows = load_claims(args.claims)
    except (OSError, ValueError) as error:
        print(f"cannot start: {error}", file=sys.stderr)
        return 2

    texts = [pad(r["claim"], args.pad_before, args.pad_after) for r in rows]
    modes = ["ungrounded", "grounded"]  # eval runs both; check runs one
    if args.command == "check":
        modes = ["grounded" if args.grounded else "ungrounded"]
    in_query = sum(claim_in_query(r["claim"], t, QUESTION) for r, t in zip(rows, texts))  # complete claim text
    if "grounded" in modes and in_query < len(texts):
        print(f"warning: {len(texts) - in_query} of {len(texts)} claims are not in their search query; "
              "the search sees the question and the first bytes of the state.", file=sys.stderr)

    with httpx.Client(headers={"Authorization": f"Bearer {api_key}"}, timeout=30.0) as client:
        try:
            results = {m: [decide(client, t, m == "grounded", args.model) for t in texts] for m in modes}
        except (DecisionsError, httpx.HTTPError) as error:
            print(f"Decisions API failed: {error}", file=sys.stderr)
            return 1

    columns = "".join(f"  {'p_' + m:>12}  {'verdict_' + m:<18}" for m in modes)
    print(f"{'id':<4} {'kind':<7} {'label':<5}{columns}".rstrip())
    for i, row in enumerate(rows):
        verdicts = [verdict(results[m][i].probability) for m in modes]
        cells = "".join(f"  {results[m][i].probability:>12.2f}  {v:<18}" for m, v in zip(modes, verdicts))
        flip = "  flip" if len(set(verdicts)) > 1 else ""
        print(f"{row['id']:<4} {row['kind']:<7} {str(row['label']).lower():<5}{cells}{flip}".rstrip())
        if args.command == "check":  # the evidence behind the verdict, one URL per line
            print("".join(f"{'':<18}{url}\n" for url in results[modes[0]][i].results), end="")
    print(f"\nclaim text in the search query: {in_query} of {len(texts)}")
    pair = {"ungrounded": "grounded", "grounded": "ungrounded"}
    summaries = [evaluate(rows, results[m], m == "grounded", args.model, results.get(pair[m])) for m in modes]
    for s in summaries:
        print(
            f"{s.mode:<10} correct {s.correct}/{s.claims}  wrong {s.wrong}  review {s.review} "
            f"({s.review / s.claims:.0%}, bound {MAX_REVIEW_RATE:.0%})  missed failures {s.missed_failures} "
            f"(bound {MAX_MISSED_FAILURES})  false contradictions {s.false_contradictions} "
            f"(bound {MAX_FALSE_CONTRADICTED_RATE:.0%})  Brier {s.brier:.3f}"
            + (f"  flips {s.flips}" if s.flips is not None else "") + "\n"
            f"{'':<10} mean input tokens {s.mean_input_tokens:.0f}  mean latency {s.mean_latency_s:.2f} s  "
            f"grounded attempts {s.grounded_attempts}  "
            + (f"mean search results {s.mean_search_results:.1f}  " if s.mode == "grounded" else "")
            + f"cost per 1,000 claims ${s.cost_per_1000:.4f}  "
            f"acceptance {'met' if accepted(s) else 'not met'}"
        )
    if "grounded" in results:
        print("grounded attempts = POSTs sent with mode set, an upper bound on billed searches.")

    OUT_DIR.mkdir(exist_ok=True)
    stamp = time.strftime("%Y%m%dT%H%M%S")
    out = OUT_DIR / f"{args.command}-before{args.pad_before}-after{args.pad_after}-{stamp}.json"
    record = {"model": args.model, "pad_before": args.pad_before, "pad_after": args.pad_after, "rows": rows,
              "summaries": [asdict(s) for s in summaries]}
    record["results"] = {m: [asdict(d) for d in r] for m, r in results.items()}
    out.write_text(json.dumps(record, indent=1), encoding="utf-8")
    print(f"saved {out}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

The full file

Save this as factcheck.py. It is the four parts above in order, separated by two blank lines.

Full code - factcheck.py

factcheck.py

"""Fact-check claims against the web with the Decisions API. One request per claim: the claim is the
`state`, one `noul` question asks whether it is true, and `mode: "web_grounding"` adds a web search."""

import argparse
import json
import os
import sys
import time
from dataclasses import asdict, dataclass, field
from pathlib import Path

import httpx

from query import claim_in_query

DECISIONS_URL = "https://api.perplexity.ai/v1/decisions"
MODEL = "pplx-decider-latest"  # selects pplx-decider-v1.2
QUESTION_NAME = "is_true"
QUESTION = {"type": "noul", "instructions": "Is the claim true?"}

SUPPORTED_MIN = 0.80  # p >= this: supported. Your code applies the cutoffs, not the model.
CONTRADICTED_MAX = 0.20  # p <= this: contradicted; anything between: review
MAX_MISSED_FAILURES = 0  # acceptance bounds the eval reports against: false claims marked supported
MAX_FALSE_CONTRADICTED_RATE = 0.05  # true claims marked contradicted, as a share of claims
MAX_REVIEW_RATE = 0.25  # share of claims sent to a person

PRICE_PER_M_INPUT = {"pplx-decider-latest": 0.02, "pplx-decider-v1.2": 0.02,  # $ per million input tokens
                     "pplx-decider-flash-latest": 0.01, "pplx-decider-flash-v1.2": 0.01}
PRICE_PER_SEARCH = 0.001  # dollars per web search when grounded
RETRIES = 3  # on 429 and 5xx; waits Retry-After seconds when it is an integer, else 1, 2, 4
MAX_WAIT_S = 30  # cap on any single wait
OUT_DIR = Path("out")

FILLER = (  # neutral text for --pad-before and --pad-after; nothing in it relates to any claim
    "The facilities team confirmed the office move for the second week of the quarter. "
    "Badge readers on the third floor will be replaced on Monday, and visitor parking "
    "moves to the lower level. The updated seating chart goes out by email once the "
    "furniture vendor confirms delivery. Please label boxes with your team name. ")


def pad(claim: str, before: int, after: int) -> str:
    """The claim on its own line, after `before` characters of FILLER and before `after` more."""
    def filler(n: int) -> list[str]:
        return [(FILLER * (n // len(FILLER) + 1))[:n]] if n else []
    return "\n".join(filler(before) + [claim] + filler(after))


class DecisionsError(Exception):
    """A Decisions API request that did not return an answer."""


@dataclass
class Decision:
    probability: float
    input_tokens: int
    latency_s: float  # wall time for the claim, retries included
    attempts: int  # POSTs sent; when grounded, each one may have run a billed search
    results: list[str] = field(default_factory=list)  # search_results URLs; [] without mode or evidence


def is_probability(p: object) -> bool:
    return isinstance(p, (int, float)) and not isinstance(p, bool) and 0.0 <= p <= 1.0


def decide(client: httpx.Client, claim: str, grounded: bool, model: str = MODEL) -> Decision:
    """POST one claim with one question. `mode` is sent only when grounded."""
    body: dict[str, object] = {"model": model, "state": claim, "questions": {QUESTION_NAME: QUESTION}}
    if grounded:
        body["mode"] = "web_grounding"
    started = time.perf_counter()
    for attempt in range(RETRIES + 1):
        response = client.post(DECISIONS_URL, json=body)
        if response.status_code == 200:
            try:
                data = response.json()
                p, tokens = data["answers"][QUESTION_NAME]["noul"], data["usage"]["input_tokens"]
                urls = [r["url"] for r in data.get("search_results") or []]
            except (ValueError, KeyError, TypeError) as error:
                raise DecisionsError(f"200 with an unexpected body: {error!r}") from None
            if not is_probability(p) or isinstance(tokens, bool) or not isinstance(tokens, int) or tokens < 0:
                raise DecisionsError(f"200 with an unexpected body: noul={p!r}, input_tokens={tokens!r}")
            return Decision(p, tokens, time.perf_counter() - started, attempt + 1, urls)
        if attempt == RETRIES or not (response.status_code == 429 or response.status_code >= 500):
            break
        retry_after = response.headers.get("Retry-After", "")
        time.sleep(min(int(retry_after) if retry_after.isdigit() else 2**attempt, MAX_WAIT_S))
    try:
        message = response.json()["error"]["message"]
    except (ValueError, KeyError, TypeError):
        message = response.text.strip()
    raise DecisionsError(f"{response.status_code}: {message}")


def verdict(p: float) -> str:
    return "supported" if p >= SUPPORTED_MIN else "contradicted" if p <= CONTRADICTED_MAX else "review"


@dataclass
class Summary:
    mode: str
    claims: int
    correct: int
    wrong: int
    review: int
    missed_failures: int  # false claims marked supported
    false_contradictions: int  # true claims marked contradicted
    flips: int | None  # None when no baseline was given
    brier: float
    mean_input_tokens: float
    mean_latency_s: float
    grounded_attempts: int  # upper bound on billed searches
    mean_search_results: float
    cost_per_1000: float


def accepted(s: Summary) -> bool:
    return (s.missed_failures <= MAX_MISSED_FAILURES and s.review <= MAX_REVIEW_RATE * s.claims
            and s.false_contradictions <= MAX_FALSE_CONTRADICTED_RATE * s.claims)


def evaluate(rows: list[dict], results: list[Decision], grounded: bool, model: str, baseline=None) -> Summary:
    """Score one mode against the labels. `baseline` is the other mode's results, for flips."""
    n = len(rows)
    for name, res in [(k, v) for k, v in (("results", results), ("baseline", baseline)) if v is not None]:
        if n == 0 or len(res) != n:
            raise ValueError(f"{name}: need one decision per row and at least one row")
        if not all(is_probability(d.probability) for d in res):
            raise ValueError(f"{name}: every probability must be a number in [0, 1]")
    if not all(isinstance(r["label"], bool) for r in rows):
        raise ValueError("every label must be true or false")
    labels = [r["label"] for r in rows]
    verdicts = [verdict(d.probability) for d in results]
    review = verdicts.count("review")
    correct = sum(v != "review" and (v == "supported") == label for v, label in zip(verdicts, labels))
    missed = sum(v == "supported" and not label for v, label in zip(verdicts, labels))
    false_contra = sum(v == "contradicted" and label for v, label in zip(verdicts, labels))
    brier = sum((d.probability - float(label)) ** 2 for d, label in zip(results, labels)) / n
    flips = None if baseline is None else sum(verdict(b.probability) != v for b, v in zip(baseline, verdicts))
    mean_tokens = sum(d.input_tokens for d in results) / n
    mean_latency = sum(d.latency_s for d in results) / n
    attempts = sum(d.attempts for d in results) if grounded else 0
    cost = 1000 * (mean_tokens * PRICE_PER_M_INPUT[model] / 1e6 + PRICE_PER_SEARCH * attempts / n)
    return Summary("grounded" if grounded else "ungrounded", n, correct, n - correct - review, review, missed,
                   false_contra, flips, brier, mean_tokens, mean_latency, attempts,
                   sum(len(d.results) for d in results) / n, cost)


def load_claims(path: str) -> list[dict]:
    """Read JSONL rows: string `id`, `claim`, `kind` and a bool `label`. Errors name the line."""
    rows = []
    for number, line in enumerate(Path(path).read_text(encoding="utf-8").splitlines(), 1):
        if not line.strip():
            continue
        try:
            row = json.loads(line)
        except ValueError as error:
            raise ValueError(f"{path}:{number}: not JSON ({error})") from None
        strings = isinstance(row, dict) and all(
            isinstance(row.get(k), str) and row[k].strip() for k in ("id", "claim", "kind"))
        if not strings or not isinstance(row.get("label"), bool):
            raise ValueError(f"{path}:{number}: needs string id, claim, kind and a true or false label")
        rows.append(row)
    if not rows:
        raise ValueError(f"{path}: no claims")
    return rows


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("command", choices=["check", "eval"])
    parser.add_argument("claims", help="JSONL with id, claim, label, kind per line")
    parser.add_argument("--grounded", action="store_true", help="check: send mode=web_grounding")
    parser.add_argument("--pad-before", type=int, default=0, help="put N chars of filler before each claim")
    parser.add_argument("--pad-after", type=int, default=0, help="put N chars of filler after each claim")
    parser.add_argument("--model", default=MODEL, choices=list(PRICE_PER_M_INPUT))
    args = parser.parse_args()
    if (args.command == "eval" and args.grounded) or min(args.pad_before, args.pad_after) < 0:
        parser.error("eval runs both modes (--grounded applies to check), and padding must be 0 or more")
    try:
        if not (api_key := os.environ.get("PERPLEXITY_API_KEY")):
            raise ValueError("set PERPLEXITY_API_KEY first")
        rows = load_claims(args.claims)
    except (OSError, ValueError) as error:
        print(f"cannot start: {error}", file=sys.stderr)
        return 2

    texts = [pad(r["claim"], args.pad_before, args.pad_after) for r in rows]
    modes = ["ungrounded", "grounded"]  # eval runs both; check runs one
    if args.command == "check":
        modes = ["grounded" if args.grounded else "ungrounded"]
    in_query = sum(claim_in_query(r["claim"], t, QUESTION) for r, t in zip(rows, texts))  # complete claim text
    if "grounded" in modes and in_query < len(texts):
        print(f"warning: {len(texts) - in_query} of {len(texts)} claims are not in their search query; "
              "the search sees the question and the first bytes of the state.", file=sys.stderr)

    with httpx.Client(headers={"Authorization": f"Bearer {api_key}"}, timeout=30.0) as client:
        try:
            results = {m: [decide(client, t, m == "grounded", args.model) for t in texts] for m in modes}
        except (DecisionsError, httpx.HTTPError) as error:
            print(f"Decisions API failed: {error}", file=sys.stderr)
            return 1

    columns = "".join(f"  {'p_' + m:>12}  {'verdict_' + m:<18}" for m in modes)
    print(f"{'id':<4} {'kind':<7} {'label':<5}{columns}".rstrip())
    for i, row in enumerate(rows):
        verdicts = [verdict(results[m][i].probability) for m in modes]
        cells = "".join(f"  {results[m][i].probability:>12.2f}  {v:<18}" for m, v in zip(modes, verdicts))
        flip = "  flip" if len(set(verdicts)) > 1 else ""
        print(f"{row['id']:<4} {row['kind']:<7} {str(row['label']).lower():<5}{cells}{flip}".rstrip())
        if args.command == "check":  # the evidence behind the verdict, one URL per line
            print("".join(f"{'':<18}{url}\n" for url in results[modes[0]][i].results), end="")
    print(f"\nclaim text in the search query: {in_query} of {len(texts)}")
    pair = {"ungrounded": "grounded", "grounded": "ungrounded"}
    summaries = [evaluate(rows, results[m], m == "grounded", args.model, results.get(pair[m])) for m in modes]
    for s in summaries:
        print(
            f"{s.mode:<10} correct {s.correct}/{s.claims}  wrong {s.wrong}  review {s.review} "
            f"({s.review / s.claims:.0%}, bound {MAX_REVIEW_RATE:.0%})  missed failures {s.missed_failures} "
            f"(bound {MAX_MISSED_FAILURES})  false contradictions {s.false_contradictions} "
            f"(bound {MAX_FALSE_CONTRADICTED_RATE:.0%})  Brier {s.brier:.3f}"
            + (f"  flips {s.flips}" if s.flips is not None else "") + "\n"
            f"{'':<10} mean input tokens {s.mean_input_tokens:.0f}  mean latency {s.mean_latency_s:.2f} s  "
            f"grounded attempts {s.grounded_attempts}  "
            + (f"mean search results {s.mean_search_results:.1f}  " if s.mode == "grounded" else "")
            + f"cost per 1,000 claims ${s.cost_per_1000:.4f}  "
            f"acceptance {'met' if accepted(s) else 'not met'}"
        )
    if "grounded" in results:
        print("grounded attempts = POSTs sent with mode set, an upper bound on billed searches.")

    OUT_DIR.mkdir(exist_ok=True)
    stamp = time.strftime("%Y%m%dT%H%M%S")
    out = OUT_DIR / f"{args.command}-before{args.pad_before}-after{args.pad_after}-{stamp}.json"
    record = {"model": args.model, "pad_before": args.pad_before, "pad_after": args.pad_after, "rows": rows,
              "summaries": [asdict(s) for s in summaries]}
    record["results"] = {m: [asdict(d) for d in r] for m, r in results.items()}
    out.write_text(json.dumps(record, indent=1), encoding="utf-8")
    print(f"saved {out}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Run it

Save test_factcheck.py and run the tests first. They need no key and no network: httpx.MockTransport stands in for the API. They cover the request body and mode, the error and retry paths, search_results in the Decision and the record, the scorecard and its bounds, and the claims file and command line with a fake decide.

test_factcheck.py

"""Offline checks for factcheck.py. No network: httpx gets a mock transport. Run: python -m pytest -q"""

import json
import sys
from pathlib import Path

import httpx
import pytest

import factcheck as fc

BAD_MODE_BODY = {"error": {"code": None, "message": 'mode must be "web_grounding" when provided',
                           "param": "mode", "type": "invalid_request_error"}}
CLAIMS_FILE = Path(__file__).with_name("claims.jsonl")
ROWS = [{"id": i, "claim": c, "label": label, "kind": k} for i, c, label, k in [
    ("a", "true one", True, "stable"), ("b", "false one", False, "stable"),
    ("c", "true two", True, "recent"), ("d", "false two", False, "recent")]]
GROUNDED = [fc.Decision(p, t, 0.4, 1, ["https://e.com/"] * u) for p, t, u in
            [(0.9, 100, 5), (0.1, 200, 0), (0.5, 300, 0), (0.85, 400, 0)]]
UNGROUNDED = [fc.Decision(0.5, 50, 0.3, 1) for _ in ROWS]


def ok_response(p: float, tokens: int, urls: list[str] | None = None) -> dict:
    body = {"model": fc.MODEL, "answers": {fc.QUESTION_NAME: {"type": "noul", "noul": p}},
            "usage": {"input_tokens": tokens, "output_tokens": 1}}
    return body if urls is None else {**body, "search_results": [{"url": u, "title": "t"} for u in urls]}


def client_with(responses: list[httpx.Response], seen: list[dict]) -> httpx.Client:
    def handle(request: httpx.Request) -> httpx.Response:  # one canned response per request; bodies recorded
        seen.append(json.loads(request.content))
        return responses.pop(0)
    return httpx.Client(transport=httpx.MockTransport(handle))


@pytest.mark.parametrize(("grounded", "model"), [(False, fc.MODEL), (True, "pplx-decider-flash-latest")])
def test_request_body_sends_mode_only_when_grounded(grounded: bool, model: str) -> None:
    seen: list[dict] = []
    urls = ["https://example.com/rust"] if grounded else None
    with client_with([httpx.Response(200, json=ok_response(0.19, 108, urls))], seen) as client:
        d = fc.decide(client, "Rust 1.0 was released in May 2015.", grounded, model)
    expected = {"model": model, "state": "Rust 1.0 was released in May 2015.",
                "questions": {fc.QUESTION_NAME: fc.QUESTION}}
    assert seen == [{**expected, "mode": "web_grounding"} if grounded else expected]
    assert (d.probability, d.input_tokens, d.attempts, d.results) == (0.19, 108, 1, urls or []) and d.latency_s >= 0


@pytest.mark.parametrize(("response", "message"), [
    (httpx.Response(400, json=BAD_MODE_BODY), '400: mode must be "web_grounding" when provided'),
    (httpx.Response(401, text="Unauthorized"), "401: Unauthorized"),
    (httpx.Response(200, json={"answers": {}}), "200 with an unexpected body"),
    (httpx.Response(200, text="<html>"), "200 with an unexpected body"),
] + [(httpx.Response(200, json=ok_response(p, tokens)), "200 with an unexpected body")
     for p, tokens in [(True, 100), (None, 100), ("0.9", 100), (1.5, 100), (0.9, -1)]]
  + [(httpx.Response(200, json={**ok_response(0.9, 100), "search_results": [{"title": "no url"}]}),
      "200 with an unexpected body")])
def test_bad_responses_raise_without_a_retry(monkeypatch, response: httpx.Response, message: str) -> None:
    monkeypatch.setattr(fc.time, "sleep", lambda _: pytest.fail("must not sleep"))
    seen: list[dict] = []
    with client_with([response], seen) as client:
        with pytest.raises(fc.DecisionsError) as info:
            fc.decide(client, "claim", grounded=True)
    assert str(info.value).startswith(message) and len(seen) == 1


@pytest.mark.parametrize(("retry_after", "wait"), [
    ("3", 3), ("Wed, 21 Oct 2026 07:28:00 GMT", 1), ("-5", 1), ("2.5", 1), ("600", fc.MAX_WAIT_S)])
def test_retry_waits_then_counts_two_attempts_and_searches(monkeypatch, retry_after: str, wait: float) -> None:
    waits: list[float] = []
    monkeypatch.setattr(fc.time, "sleep", waits.append)
    responses = [httpx.Response(429, headers={"Retry-After": retry_after}, json={"error": {"message": "slow"}}),
                 httpx.Response(200, json=ok_response(0.5, 100))]
    with client_with(responses, []) as client:
        d = fc.decide(client, "claim", grounded=True)
    assert (d.probability, d.attempts, waits) == (0.5, 2, [wait])
    assert fc.evaluate(ROWS[:1], [d], grounded=True, model=fc.MODEL).grounded_attempts == 2


def test_gives_up_after_retries_on_5xx(monkeypatch) -> None:
    waits: list[float] = []
    monkeypatch.setattr(fc.time, "sleep", waits.append)
    seen: list[dict] = []
    with client_with([httpx.Response(503, text="upstream") for _ in range(fc.RETRIES + 1)], seen) as client:
        with pytest.raises(fc.DecisionsError, match="503: upstream"):
            fc.decide(client, "claim", grounded=False)
    assert (len(seen), waits) == (fc.RETRIES + 1, [1, 2, 4])  # exponential backoff without Retry-After


@pytest.mark.parametrize(("p", "expected"), [
    (1.0, "supported"), (0.80, "supported"), (0.7999, "review"), (0.2001, "review"), (0.20, "contradicted")])
def test_verdict_cutoffs_are_inclusive(p: float, expected: str) -> None:
    assert fc.verdict(p) == expected


def test_evaluate_scorecard() -> None:
    s = fc.evaluate(ROWS, GROUNDED, grounded=True, model=fc.MODEL, baseline=UNGROUNDED)
    assert (s.correct, s.wrong, s.review) == (2, 1, 1)
    assert (s.missed_failures, s.false_contradictions) == (1, 0)  # d: false, 0.85 -> supported
    assert s.flips == 3  # a, b, d changed verdict; c stayed in review
    assert s.brier == pytest.approx((0.01 + 0.01 + 0.25 + 0.7225) / 4)


def test_evaluate_cost_adds_one_search_per_grounded_attempt() -> None:
    grounded = fc.evaluate(ROWS, GROUNDED, grounded=True, model=fc.MODEL)
    ungrounded = fc.evaluate(ROWS, UNGROUNDED, grounded=False, model=fc.MODEL)
    flash = fc.evaluate(ROWS, UNGROUNDED, grounded=False, model="pplx-decider-flash-latest")
    assert (grounded.mean_input_tokens, grounded.mean_latency_s) == (250, pytest.approx(0.4))
    assert (grounded.grounded_attempts, ungrounded.grounded_attempts, ungrounded.flips) == (4, 0, None)
    assert (grounded.mean_search_results, ungrounded.mean_search_results) == (1.25, 0)
    # 1,000 claims x 250 tokens x $0.02/M = $0.005, plus 1,000 searches x $0.001 = $1.00
    assert grounded.cost_per_1000 == pytest.approx(1.005)
    assert ungrounded.cost_per_1000 == pytest.approx(1000 * 50 * 0.02 / 1e6)
    assert flash.cost_per_1000 == pytest.approx(1000 * 50 * 0.01 / 1e6)


@pytest.mark.parametrize(("rows", "results", "baseline"), [
    ([], [], None),
    (ROWS, GROUNDED[:3], None),
    ([{**ROWS[0], "label": "true"}], GROUNDED[:1], None),
    (ROWS[:1], [fc.Decision(1.5, 100, 0.1, 1)], None),
    (ROWS[:1], [fc.Decision(float("nan"), 100, 0.1, 1)], None),
    (ROWS, GROUNDED, UNGROUNDED[:3]),
    (ROWS[:1], GROUNDED[:1], [fc.Decision(-0.1, 100, 0.1, 1)]),
])
def test_evaluate_rejects_bad_input(rows: list[dict], results: list[fc.Decision], baseline) -> None:
    with pytest.raises(ValueError):
        fc.evaluate(rows, results, grounded=False, model=fc.MODEL, baseline=baseline)


@pytest.mark.parametrize(("true_p", "false_p", "counts", "passes"), [
    (0.0, 0.0, (0, 13, 0), False),  # contradict everything
    (0.5, 0.5, (0, 0, 24), False),  # review everything
    (0.98, 0.0, (0, 0, 0), True)])  # the recorded grounded run's shape
def test_acceptance_needs_all_three_bounds(true_p: float, false_p: float, counts: tuple, passes: bool) -> None:
    rows = fc.load_claims(str(CLAIMS_FILE))
    results = [fc.Decision(true_p if r["label"] else false_p, 1000, 0.4, 1) for r in rows]
    s = fc.evaluate(rows, results, grounded=True, model=fc.MODEL)
    assert (s.missed_failures, s.false_contradictions, s.review) == counts
    assert fc.accepted(s) is passes


@pytest.mark.parametrize(("false_contradictions", "reviews", "passes"), [
    (1, 0, True), (2, 0, False),  # 1 of 20 is the 5% bound exactly; 2 is over it
    (0, 5, True), (0, 6, False),  # 5 of 20 is the 25% review bound exactly
])
def test_acceptance_bounds_are_inclusive(false_contradictions: int, reviews: int, passes: bool) -> None:
    rows = [{"id": f"t{i}", "claim": "c", "label": True, "kind": "stable"} for i in range(20)]
    ps = [0.0] * false_contradictions + [0.5] * reviews
    ps += [1.0] * (20 - len(ps))
    s = fc.evaluate(rows, [fc.Decision(p, 100, 0.1, 1) for p in ps], grounded=False, model=fc.MODEL)
    assert (s.false_contradictions, s.review) == (false_contradictions, reviews)
    assert fc.accepted(s) is passes


@pytest.mark.parametrize(("content", "where"), [
    ("", "no claims"),
    ('{"id": "a", "claim": "ok", "label": true, "kind": "stable"}\n'
     '{"id": "b", "label": true, "kind": "stable"}\n', ":2:"),
    ('{"id": "a", "claim": "ok", "label": "true", "kind": "stable"}\n', ":1:"),
    ("not json\n", ":1: not JSON"),
])
def test_load_claims_names_the_bad_line(tmp_path: Path, content: str, where: str) -> None:
    (tmp_path / "claims.jsonl").write_text(content, encoding="utf-8")
    with pytest.raises(ValueError, match=where):
        fc.load_claims(str(tmp_path / "claims.jsonl"))


def test_pad_puts_filler_on_its_own_lines() -> None:
    claim = "Python 3.12 was released in October 2023."
    assert fc.pad(claim, 0, 0) == claim
    assert fc.pad(claim, 600, 0) == fc.pad(claim, 600, 600).rsplit("\n", 1)[0]
    assert len(fc.pad(claim, 600, 0)) == len(claim) + 601
    assert fc.pad(claim, 0, 5).splitlines() == [claim, "The f"] and fc.pad(claim, 3, 0).splitlines() == ["The", claim]


@pytest.fixture
def cli(monkeypatch, tmp_path: Path):
    monkeypatch.chdir(tmp_path)
    monkeypatch.setenv("PERPLEXITY_API_KEY", "k")

    def run(*args: str) -> int:
        monkeypatch.setattr(sys, "argv", ["factcheck.py", *args])
        return fc.main()
    return run


def test_cli_stops_before_any_request_on_bad_input(cli, monkeypatch, tmp_path: Path, capsys) -> None:
    monkeypatch.setattr(fc, "decide", lambda *a, **k: pytest.fail("must not call the API"))
    with pytest.raises(SystemExit):
        cli("eval", str(CLAIMS_FILE), "--grounded")
    with pytest.raises(SystemExit):
        cli("check", str(CLAIMS_FILE), "--pad-after", "-1")
    (tmp_path / "bad.jsonl").write_text('{"id": "a"}\n', encoding="utf-8")
    assert cli("check", str(tmp_path / "bad.jsonl")) == 2 and "bad.jsonl:1:" in capsys.readouterr().err
    monkeypatch.delenv("PERPLEXITY_API_KEY")
    assert cli("check", str(CLAIMS_FILE)) == 2 and "PERPLEXITY_API_KEY" in capsys.readouterr().err


def test_cli_check_and_eval_runs(cli, monkeypatch, tmp_path: Path, capsys) -> None:
    rows = fc.load_claims(str(CLAIMS_FILE))
    by_text = {t: r for r in rows for t in (r["claim"], fc.pad(r["claim"], 600, 0))}

    def fake_decide(client, text, grounded, model=fc.MODEL) -> fc.Decision:
        row = by_text[text]
        if not grounded and row["id"] == "s08":  # a false claim marked supported
            return fc.Decision(0.85, 200, 0.2, 1)
        urls = [f"https://example.com/{row['id']}"] if grounded else []
        return fc.Decision(0.98 if row["label"] else 0.0, 1800 if grounded else 200, 0.3, 1, urls)

    monkeypatch.setattr(fc, "decide", fake_decide)
    assert cli("check", str(CLAIMS_FILE), "--grounded") == 0
    out = capsys.readouterr().out
    assert "p_grounded" in out and "p_ungrounded" not in out and "grounded attempts 24" in out
    assert "claim text in the search query: 24 of 24" in out and "acceptance met" in out
    assert "mean search results 1.0" in out and out.count("https://example.com/") == 24  # one URL line per claim
    assert cli("eval", str(CLAIMS_FILE), "--pad-before", "600") == 0  # the warning fires, ungrounded fails the gate
    captured = capsys.readouterr()
    assert "warning: 24 of 24 claims are not in their search query" in captured.err
    assert "claim text in the search query: 0 of 24" in captured.out and "https://example.com/" not in captured.out
    assert "missed failures 1 (bound 0)" in captured.out and "acceptance not met" in captured.out
    record = json.loads(next((tmp_path / "out").glob("eval-before600-after0-*.json")).read_text(encoding="utf-8"))
    assert [s["mode"] for s in record["summaries"]] == ["ungrounded", "grounded"]
    first = record["results"]["grounded"][0]
    assert (first["attempts"], first["results"]) == (1, ["https://example.com/s01"])

    def failing_decide(client, text, grounded, model=fc.MODEL) -> fc.Decision:
        raise fc.DecisionsError("503: upstream")

    monkeypatch.setattr(fc, "decide", failing_decide)
    assert cli("check", str(CLAIMS_FILE)) == 1 and "Decisions API failed: 503" in capsys.readouterr().err

Run the tests

python -m pytest -q

........................................................................ [ 86%]
...........                                                              [100%]
83 passed in 0.19s

Then check the claims with grounding on. check prints one probability and one verdict per claim with the URLs the model saw under it, the query count, the summary for that mode, and the path of the JSON it saved. Every output on this page was recorded on 2026-11-04 with pplx-decider-latest.

Check the claims

python factcheck.py check claims.jsonl --grounded

id   kind    label    p_grounded  verdict_grounded
s01  stable  true           1.00  supported
                  https://docs.python.org/3/whatsnew/3.12.html
                  https://lenz.io/c/python-3-12-release-october-2-2023-4e83fd9d
                  https://www.python.org/downloads/release/python-3120/
                  https://www.python.org/doc/versions/
                  https://docs.python.org/3.12/whatsnew/changelog.html
s02  stable  true           1.00  supported
                  https://safeguard.sh/resources/blog/rust-release-history-from-1-0-to-the-2024-edition-six-week-trains-editions-governance-and-security-response
                  https://www.zdnet.com/article/rust-turns-10-how-a-broken-elevator-changed-software-forever/
                  https://en.wikipedia.org/wiki/Rust_(programming_language)
                  https://blog.rust-lang.org/2015/05/15/Rust-1.0/
                  https://doc.rust-lang.org/edition-guide/rust-2015/index.html
s03  stable  true           1.00  supported
                  https://wwf.org.au/what-we-do/oceans/great-barrier-reef
                  https://oceanservice.noaa.gov/facts/gbrlargeststructure.html
                  https://en.wikipedia.org/wiki/Great_Barrier_Reef
                  https://www.museum.qld.gov.au/learn-and-discover/queenslands-environments-and-biodiversity/the-great-barrier-reef/
                  https://www.gbrmpa.gov.au/learn/fascinating-facts-about-great-barrier-reef
s04  stable  true           1.00  supported
                  https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status
                  https://developers.cloudflare.com/support/troubleshooting/http-status-codes/4xx-client-error/error-404/
                  https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/404
                  https://www.context.dev/glossary/http-errors/404
                  https://404meanings.com/404-meaning/
s05  stable  false          0.00  contradicted
                  https://en.wikipedia.org/wiki/History_of_Linux
                  https://www.redhat.com/en/blog/celebrating-30-years-linux-kernel-and-gplv2
                  https://www.linuxfoundation.org/blog/blog/anniversary-of-first-linux-kernel-release-a-look-at-collaborative-value
                  https://opensource.com/life/15/8/happy-24th-birthday-linux-kernel
                  https://lwn.net/1999/features/1998timeline/
s06  stable  false          0.00  contradicted
                  https://www.britannica.com/story/7-or-8-summits-the-worlds-highest-mountains-by-continent
                  https://en.tempo.co/read/2118799/7-highest-mountains-in-the-world-one-from-each-continent
                  https://www.il.tzembassy.go.tz/resources/view/12-interesting-facts-about-mount-kilimanjaro
                  https://www.nepalhikingteam.com/seven-summits-highest-peaks-by-continent
                  https://thebucketlistcompany.co.uk/8-mount-kilimanjaro-facts-to-wow-your-fellow-climbers/
s07  stable  false          0.00  contradicted
                  https://en.wikipedia.org/wiki/Guido_van_Rossum
                  https://www.devclass.com/development/2025/04/11/20-years-of-git-never-a-big-thing-for-me-says-inventor-linus-torvalds/1620325
                  https://about.gitlab.com/blog/celebrating-gits-20th-anniversary-with-creator-linus-torvalds/
                  https://www.welcometothejungle.com/en/articles/btc-history-git
                  https://www.geeksacademy.it/en-en/article-83/who-is-the-father-of-python-one-of-the-most-used-programming-languages-in-the-world
s08  stable  false          0.00  contradicted
                  https://www.postgresql.org/about/licence/?lang=en
                  https://www.postgresql.org/about/press/faq/
                  https://wiki.postgresql.org/wiki/FAQ
                  https://en.wikipedia.org/wiki/PostgreSQL
                  https://news.ycombinator.com/item?id=25932194
r01  recent  true           1.00  supported
                  https://www.nytimes.com/live/2026/10/09/world/nobel-peace-prize-winner-2026/heres-the-latest
                  https://www.cnn.com/2026/10/09/world/live-news/nobel-peace-prize-2026-winner-intl
                  https://www.bbc.com/news/articles/cm9wz5kng0x1o
                  https://news.un.org/en/story/2026/10/1168558
                  https://www.nobelprize.org/educational/nobel-prize-lessons/text-for-students-peace-2026/
r02  recent  true           1.00  supported
                  https://physics.berkeley.edu/news/berkeley-helped-make-icecube-success%E2%80%94and-earn-its-leader-2026-nobel-prize
                  https://www.reuters.com/science/francis-halzen-wins-2026-nobel-prize-physics-2026-10-06/
                  https://www.opb.org/article/2026/10/06/francis-halzen-wins-nobel-prize-in-physics-for-work-on-high-energy-neutrinos/
                  https://www.nature.com/articles/d41586-026-03190-0?error=cookies_not_supported&code=32e41dd9-8c59-473b-a396-d743165705c8
                  https://www.bbc.com/news/articles/cq203mymlvkeo
r03  recent  true           1.00  supported
                  https://www.reuters.com/5f374b32b07b/lifestyle/canadian-writer-anne-carson-wins-2026-nobel-prize-literature-2026-10-08/
                  https://www.npr.org/2026/10/08/nx-s1-5984350/nobel-prize-in-literature-anne-carson
                  https://ca.rollingstone.com/culture/canadian-anne-carson-wins-nobel-prize/
                  https://www.cnn.com/2026/10/08/style/nobel-prize-literature-anne-carson-winner-2026-intl
                  https://amp.dw.com/en/nobel-prize-in-literature-goes-to-canadas-anne-carson/a-79594136
r04  recent  true           1.00  supported
                  https://link.springer.com/article/10.1007/s11224-026-02931-1
                  https://www.cnrs.fr/en/press/2026-nobel-prize-chemistry-awarded-henri-kagan-his-work-asymmetric-catalysis
                  https://www.nobelprize.org/prizes/chemistry/2026/popular-information/
                  https://www.nobelprize.org/prizes/chemistry/2026/summary/
                  https://www.cnn.com/2026/10/07/science/nobel-prize-chemistry-2026-henri-kagan-kenso-soai-winner-intl
r05  recent  true           1.00  supported
                  https://www.cnn.com/2026/10/28/sport/live-news/game-5-world-series
                  https://www.mlb.com/brewers/news/brewers-beat-rays-game-5-win-2026-world-series
                  https://www.mlb.com/brewers/news/brewers-win-world-series-2026
                  https://www.espn.com/mlb/story/_/id/50294394/world-series-brewers-champions-franchise-first-title
                  https://www.usatoday.com/story/sports/mlb/2026/10/29/brewers-first-world-series-title-mlb-teams-never-won-world-series/92490734007/
r06  recent  true           1.00  supported
                  https://blog.python.org/2026/10/python-3150-final-is-here/
                  https://docs.python.org/3/whatsnew/changelog.html
                  https://realpython.com/python-news-november-2026/
                  https://xt.pt/software/python-3-15-0-ships-eight-days-late-and-the-lazy-import-fixes-are-still-coming
                  https://realpython.com/python-news-october-2026/
r07  recent  true           1.00  supported
                  https://global.espn.com/football/story/_/id/50236983/harry-kane-2026-ballon-dor-kylian-mbappe-lamine-yamal
                  https://www.nytimes.com/athletic/7710586/2026/10/26/harry-kane-ballon-dor-winner-2026/
                  https://www.premierleague.com/en/news/4749936/harry-kane-wins-ballon-dor-as-premier-league-stars-learn-rankings
                  https://www.skysports.com/football/news/13601934/harry-kane-wins-2026-ballon-dor-bayern-munich-striker-becomes-first-english-winner-of-award-in-25-years
                  https://www.uefa.com/uefachampionsleague/news/02aa-21ec47378e70-b29defb43769-1000--harry-kane-wins-2026-men-s-ballon-d-or/
r08  recent  true           1.00  supported
                  https://www.federalreserve.gov/newsevents/pressreleases/monetary20261028a1.htm
                  https://www.federalreserve.gov/monetarypolicy/files/monetary20261028a1.pdf
                  https://www.federalreserve.gov/newsevents/pressreleases/monetary20261028a.htm?
                  https://www.cnbc.com/2026/10/28/fed-meeting-today-live-updates.html
                  https://finance.yahoo.com/economy/policy/articles/fed-holds-rates-mortgage-relief-185653601.html
r09  recent  false          0.00  contradicted
                  https://blog.pureinventionbook.com/p/a-wild-prize-chase
                  https://indianexpress.com/article/books-and-literature/nobel-prize-in-literature-2026-live-updates-classicist-poet-or-chinas-dream-weaver-anne-carson-and-can-xue-go-head-to-head-on-bookies-lists-10912371/
                  https://note.com/tsuyoshin38/n/n3e74313c9373?hl=en
                  https://www.britannica.com/biography/Haruki-Murakami
                  https://believermagazine.substack.com/p/les-nobelisables-haruki-murakami
r10  recent  false          0.00  contradicted
                  https://www.nbcnews.com/sports/mlb/milwaukee-brewers-tampa-bay-rays-world-series-game-5-result-rcna605974
                  https://www.bbc.com/sport/baseball/articles/c5x2v991l0lvo
                  https://sports.yahoo.com/mlb/live/brewers-rays-world-series-game-5-box-score-live-updates-lineups-starting-pitchers-highlights-where-to-watch-230000010.html
                  https://sports.yahoo.com/mlb/article/world-series-2026-for-the-rays-being-the-best-team-in-the-american-league-simply-wasnt-enough-074822560.html
                  https://www.philstar.com/campus/sports/2026/10/30/2566690/long-wait-finally-over
r11  recent  false          0.00  contradicted
                  https://www.espn.com/soccer/story/_/id/50295834/why-real-madrid-kylian-mbappe-barcelona-lamine-yamal-lost-ballon-dor-harry-kane
                  https://global.espn.com/football/story/_/id/50289230/ballon-dor-2026-winners-blunder-harry-kane-lamine-yamal-kylian-mbappe
                  https://www.espn.com/soccer/story/_/id/50289230/ballon-dor-2026-winners-blunder-harry-kane-lamine-yamal-kylian-mbappe
                  https://global.espn.com/football/story/_/id/50236983/harry-kane-2026-ballon-dor-kylian-mbappe-lamine-yamal
                  https://www.cbssports.com/soccer/news/harry-kane-alexia-putellas-win-2026-ballon-dor-awards/live/
r12  recent  false          0.00  contradicted
                  https://www.federalreserve.gov/newsevents/pressreleases/monetary20261028a1.htm
                  https://www.federalreserve.gov/monetarypolicy/files/monetary20261028a1.pdf
                  https://www.jpmorgan.com/insights/markets-and-economy/economy/kevin-warsh-october-2026-federal-reserve-holds-rates-vows-to-deliver-price-stability
                  https://www.cnbc.com/2026/10/28/fed-meeting-today-live-updates.html
                  https://www.courant.com/2026/10/28/fed-meeting-interest-rates/
r13  recent  false          0.00  contradicted
                  https://www.apple.com/newsroom/2026/09/apple-debuts-iphone-18-pro-and-iphone-18-pro-max/
                  https://www.macworld.com/article/672537/when-is-apples-next-event.html
                  https://www.theapplepost.com/2026/09/10/72070/apple-event-rumors-vs-reality-what-the-leaks-got-right-what-they-got-wrong-and-the-biggest-surprises/
                  https://www.bytepreview.com/en/articulos/no-confirmed-phone-launches-in-october-2026-what-is-really-happening
                  https://www.apple.com/newsroom/2026/09/the-latest-iphone-apple-watch-and-airpods-lineups-arrive-in-stores-worldwide/
r14  recent  false          0.00  contradicted
                  https://www.nobelprize.org/prizes/physics/2026/prize-announcement/
                  https://www.nobelprize.org/prizes/physics/2026/summary/
                  https://www.nobelprize.org/prizes/physics/2026/advanced-information/
                  https://spectator.com/article/why-im-sceptical-about-a-superconductor-breakthrough/
                  https://www.nobelprize.org/physics-prize-2/
r15  recent  false          0.00  contradicted
                  https://www.nobelprize.org/prizes/economic-sciences/all-prizes-in-economic-sciences/
                  https://www.nobelprize.org/prizes/economic-sciences/
                  https://www.nobelprize.org/prizes/lists/all-prizes-in-economic-sciences/all/
                  https://www.nber.org/nber-news/nobel-laureates
                  https://www.nobelprize.org/prizes/economic-sciences/2024/summary/summary/
r16  recent  true           1.00  supported
                  https://www.cbc.ca/news/world/nobel-prize-sweden-medicine-physiology-9.7366536
                  https://www.nobelprize.org/prizes/medicine/2026/popular-information/
                  https://www.nbcnews.com/world/europe/nobel-prize-medicine-2026-karl-deisseroth-peter-hegemann-georg-nagel-rcna601564
                  https://www.nobelprize.org/prizes/medicine/2026/summary/
                  https://www.smithsonianmag.com/smart-news/trio-of-scientists-wins-medicine-nobel-for-a-method-to-turn-nerve-cells-on-and-off-with-light-helping-to-unravel-the-brains-mysteries-180989618/

claim text in the search query: 24 of 24
grounded   correct 24/24  wrong 0  review 0 (0%, bound 25%)  missed failures 0 (bound 0)  false contradictions 0 (bound 5%)  Brier 0.000
           mean input tokens 1908  mean latency 0.50 s  grounded attempts 24  mean search results 5.0  cost per 1,000 claims $1.0382  acceptance met
grounded attempts = POSTs sent with mode set, an upper bound on billed searches.
saved out/check-before0-after0-20261104T174406.json

Drop --grounded to get the probability without a search.

Measure it

Run eval on the same file. It sends every claim twice, without mode and with it, and lines the two up; the rest of this page reads from it.

Run both modes

python factcheck.py eval claims.jsonl

Observed output

id   kind    label  p_ungrounded  verdict_ungrounded    p_grounded  verdict_grounded
s01  stable  true           1.00  supported                   1.00  supported
s02  stable  true           1.00  supported                   1.00  supported
s03  stable  true           1.00  supported                   1.00  supported
s04  stable  true           1.00  supported                   1.00  supported
s05  stable  false          0.04  contradicted                0.00  contradicted
s06  stable  false          0.00  contradicted                0.00  contradicted
s07  stable  false          0.00  contradicted                0.00  contradicted
s08  stable  false          0.15  contradicted                0.00  contradicted
r01  recent  true           0.01  contradicted                1.00  supported           flip
r02  recent  true           0.04  contradicted                1.00  supported           flip
r03  recent  true           0.04  contradicted                1.00  supported           flip
r04  recent  true           0.04  contradicted                1.00  supported           flip
r05  recent  true           0.10  contradicted                1.00  supported           flip
r06  recent  true           0.07  contradicted                1.00  supported           flip
r07  recent  true           0.13  contradicted                1.00  supported           flip
r08  recent  true           0.24  review                      1.00  supported           flip
r09  recent  false          0.02  contradicted                0.00  contradicted
r10  recent  false          0.04  contradicted                0.00  contradicted
r11  recent  false          0.18  contradicted                0.00  contradicted
r12  recent  false          0.10  contradicted                0.00  contradicted
r13  recent  false          0.03  contradicted                0.00  contradicted
r14  recent  false          0.00  contradicted                0.00  contradicted
r15  recent  false          0.03  contradicted                0.00  contradicted
r16  recent  true           0.02  contradicted                1.00  supported           flip

claim text in the search query: 24 of 24
ungrounded correct 15/24  wrong 8  review 1 (4%, bound 25%)  missed failures 0 (bound 0)  false contradictions 8 (bound 5%)  Brier 0.326  flips 9
           mean input tokens 99  mean latency 0.43 s  grounded attempts 0  cost per 1,000 claims $0.0020  acceptance not met
grounded   correct 24/24  wrong 0  review 0 (0%, bound 25%)  missed failures 0 (bound 0)  false contradictions 0 (bound 5%)  Brier 0.000  flips 9
           mean input tokens 1906  mean latency 0.58 s  grounded attempts 24  mean search results 5.0  cost per 1,000 claims $1.0381  acceptance met
grounded attempts = POSTs sent with mode set, an upper bound on billed searches.
saved out/eval-before0-after0-20261104T174430.json

Both modes got the eight stable claims right with probabilities at 1.00 or at 0.15 and under. The 16 recent claims split the two modes. Ungrounded, the 9 true recent claims printed between 0.01 and 0.24: 8 were marked contradicted and one, r08, went to review. The 7 false recent claims also came back low, 0.00 to 0.18, which counts as correct, but it is the same behavior: without a search, every claim about late 2026 got a low probability, true or false. That is 15 of 24 correct, but 8 of the 9 true recent claims were marked contradicted, which fails the false-contradiction bound of 5%: acceptance not met. Grounded, the same 24 requests printed 1.00 for every true claim and 0.00 for every false one. 24 of 24, no reviews, no wrong verdicts, all three bounds met, and 9 flips, all of them true recent claims moving to supported. r01 is the Navi Pillay claim from the top of the page: 0.01 to 1.00. Mean input tokens went from 99 to 1,906, because the evidence is now part of the state; individual grounded requests ranged from 1,213 to 3,143 tokens, with 5 results for every claim. Mean latency went from 0.43 s to 0.58 s in this run; the earlier check run averaged 0.50 s grounded, so the latency figures describe this session more than the mode.

What the search sees

Two runs with the same 24 claims and the same 600 characters of FILLER, which differ in where the claim sits. --pad-before 600 puts the filler first and the claim on the line after it; --pad-after 600 puts the claim first and the filler after it. Moving the claim changes two things at once: the search prefix, and the order of the state the model reads. claim_in_query says the claim is outside every query in the first run and inside every query in the second, and the script warns before the first run sends anything:

Run both paddings

python factcheck.py eval claims.jsonl --pad-before 600
python factcheck.py eval claims.jsonl --pad-after 600

warning: 24 of 24 claims are not in their search query; the search sees the question and the first bytes of the state.

Observed output: claim after the filler

id   kind    label  p_ungrounded  verdict_ungrounded    p_grounded  verdict_grounded
s01  stable  true           0.99  supported                   0.92  supported
s02  stable  true           0.75  review                      0.15  contradicted        flip
s03  stable  true           1.00  supported                   1.00  supported
s04  stable  true           0.99  supported                   0.99  supported
s05  stable  false          0.07  contradicted                0.03  contradicted
s06  stable  false          0.00  contradicted                0.00  contradicted
s07  stable  false          0.00  contradicted                0.00  contradicted
s08  stable  false          0.95  supported                   0.82  supported
r01  recent  true           0.02  contradicted                0.07  contradicted
r02  recent  true           0.26  review                      0.21  review
r03  recent  true           0.33  review                      0.20  review
r04  recent  true           0.21  review                      0.26  review
r05  recent  true           0.07  contradicted                0.08  contradicted
r06  recent  true           0.28  review                      0.07  contradicted        flip
r07  recent  true           0.07  contradicted                0.12  contradicted
r08  recent  true           0.39  review                      0.17  contradicted        flip
r09  recent  false          0.10  contradicted                0.10  contradicted
r10  recent  false          0.05  contradicted                0.07  contradicted
r11  recent  false          0.10  contradicted                0.21  review              flip
r12  recent  false          0.21  review                      0.11  contradicted        flip
r13  recent  false          0.17  contradicted                0.05  contradicted
r14  recent  false          0.01  contradicted                0.03  contradicted
r15  recent  false          0.13  contradicted                0.16  contradicted
r16  recent  true           0.12  contradicted                0.23  review              flip

claim text in the search query: 0 of 24
ungrounded correct 12/24  wrong 5  review 7 (29%, bound 25%)  missed failures 1 (bound 0)  false contradictions 4 (bound 5%)  Brier 0.294  flips 6
           mean input tokens 212  mean latency 0.30 s  grounded attempts 0  cost per 1,000 claims $0.0042  acceptance not met
grounded   correct 12/24  wrong 7  review 5 (21%, bound 25%)  missed failures 1 (bound 0)  false contradictions 6 (bound 5%)  Brier 0.330  flips 6
           mean input tokens 915  mean latency 0.44 s  grounded attempts 24  mean search results 5.0  cost per 1,000 claims $1.0183  acceptance not met
grounded attempts = POSTs sent with mode set, an upper bound on billed searches.
saved out/eval-before600-after0-20261104T174507.json

Observed output: claim before the filler

id   kind    label  p_ungrounded  verdict_ungrounded    p_grounded  verdict_grounded
s01  stable  true           0.99  supported                   1.00  supported
s02  stable  true           0.99  supported                   1.00  supported
s03  stable  true           1.00  supported                   1.00  supported
s04  stable  true           1.00  supported                   0.99  supported
s05  stable  false          0.05  contradicted                0.00  contradicted
s06  stable  false          0.00  contradicted                0.00  contradicted
s07  stable  false          0.00  contradicted                0.00  contradicted
s08  stable  false          0.23  review                      0.00  contradicted        flip
r01  recent  true           0.01  contradicted                1.00  supported           flip
r02  recent  true           0.03  contradicted                1.00  supported           flip
r03  recent  true           0.03  contradicted                1.00  supported           flip
r04  recent  true           0.06  contradicted                1.00  supported           flip
r05  recent  true           0.03  contradicted                1.00  supported           flip
r06  recent  true           0.06  contradicted                1.00  supported           flip
r07  recent  true           0.04  contradicted                1.00  supported           flip
r08  recent  true           0.24  review                      1.00  supported           flip
r09  recent  false          0.01  contradicted                0.00  contradicted
r10  recent  false          0.01  contradicted                0.00  contradicted
r11  recent  false          0.09  contradicted                0.00  contradicted
r12  recent  false          0.09  contradicted                0.00  contradicted
r13  recent  false          0.01  contradicted                0.01  contradicted
r14  recent  false          0.00  contradicted                0.00  contradicted
r15  recent  false          0.03  contradicted                0.00  contradicted
r16  recent  true           0.02  contradicted                0.99  supported           flip

claim text in the search query: 24 of 24
ungrounded correct 14/24  wrong 8  review 2 (8%, bound 25%)  missed failures 0 (bound 0)  false contradictions 8 (bound 5%)  Brier 0.338  flips 10
           mean input tokens 212  mean latency 0.40 s  grounded attempts 0  cost per 1,000 claims $0.0042  acceptance not met
grounded   correct 24/24  wrong 0  review 0 (0%, bound 25%)  missed failures 0 (bound 0)  false contradictions 0 (bound 5%)  Brier 0.000  flips 10
           mean input tokens 1602  mean latency 0.52 s  grounded attempts 24  mean search results 5.0  cost per 1,000 claims $1.0320  acceptance met
grounded attempts = POSTs sent with mode set, an upper bound on billed searches.
saved out/eval-before0-after600-20261104T174530.json

With the claim after the filler, grounded accuracy fell from 24 of 24 to 12 of 24, with 5 claims in review and 5 of the 9 true recent claims marked contradicted; every search_results list in that run holds office-move pages. Even s02, “Rust 1.0 was released in May 2015”, which both modes had at 1.00 unpadded, printed 0.15. With the claim before the filler, the grounded run got 24 of 24 and met all three bounds, the same verdicts as the unpadded run. The experiment does not separate the two effects: the first run changed both the evidence and the order of the state, and the ungrounded results moved too, 12 of 24 with the claim after the filler and 14 of 24 with it before, against 15 of 24 unpadded. s08, “PostgreSQL is licensed under the GNU General Public License”, went from 0.15 unpadded to 0.95 ungrounded with the filler in front of it, a missed failure, and 0.82 grounded. What the recorded outputs do show is that the claim’s position decided what the search returned, and the run with the claim first is the one that met the bounds. To see the wrong evidence with your own eyes, run check on one buried claim. The URLs under the row are the pages the search returned for a query that began with the office move:

Check one buried claim

grep '"r01"' claims.jsonl > r01.jsonl
python factcheck.py check r01.jsonl --grounded --pad-before 600

warning: 1 of 1 claims are not in their search query; the search sees the question and the first bytes of the state.
id   kind    label    p_grounded  verdict_grounded
r01  recent  true           0.07  contradicted
                  https://kadence.co/templates/office-move-checklist/
                  https://www.deskbird.com/blog/product-updates-q3-2026
                  https://moving-company-guide.com/office-relocation/office-relocation-checklist
                  https://nilsonvan.com/blog/office-move-checklist/
                  https://www.goborntomove.com/blogs/office-moving-checklist-week-by-week-plan

claim text in the search query: 0 of 1
grounded   correct 0/1  wrong 1  review 0 (0%, bound 25%)  missed failures 0 (bound 0)  false contradictions 1 (bound 5%)  Brier 0.865
           mean input tokens 913  mean latency 0.65 s  grounded attempts 1  mean search results 5.0  cost per 1,000 claims $1.0183  acceptance not met
grounded attempts = POSTs sent with mode set, an upper bound on billed searches.
saved out/check-before600-after0-20261104T174530.json

Read the numbers

Recent against stable. Grounding changed no verdict on the 8 stable claims. If your claims are about things settled years ago, measure before you pay for the search.
A low ungrounded probability is not a contradiction. Ungrounded, the 9 true recent claims printed between 0.01 and 0.24 and the 7 false ones between 0.00 and 0.18. Nothing in those numbers separates “false” from “no evidence either way”. Treat contradicted from an ungrounded request as “unknown” for claims about recent events.
Evidence about the wrong thing looks like evidence. With the claim buried, every grounded response carried office-move URLs and a verdict that looked as definite as any other; search_results is the only place in the response that shows what the evidence was. Keep the URLs in your records, as out/ does here, and look at them when a verdict surprises you.
Grounding fails open, and the response says so. This recipe sends one question, so one search per request: if it fails, runs out of time (1 second per search, 3 seconds for the request), or returns nothing, the model answers from your original state and search_results is []. With several questions, the searches that succeed are used and search_results is [] only when none return. The script records the URLs per claim and prints mean search results on the grounded line, so a run where evidence went missing shows up in the summary; in the recorded runs it was 5.0 every time, 5 results for every claim.
Run-to-run drift. We ran eval twice, back to back. The second run held all 48 verdicts. Ungrounded, four printed probabilities changed by 0.01 at two decimals (r02 and r04 from 0.04 to 0.03, r05 and r12 from 0.10 to 0.11), seven more changed by less than a hundredth, and 13 were identical to the last digit; grounded, none changed at two decimals and 12 were identical. Mean latency was 0.30 s and 0.49 s against 0.43 s and 0.58 s. The nearest call was r11 ungrounded at 0.18, two hundredths under the contradicted cutoff; a claim that settles that close to a cutoff can print a different verdict on another day, which the ticket triage recipe saw happen.
Probabilities are the model’s estimate. 24 of 24 is one recorded run, not a guarantee. The verdict is only as good as the evidence the search found and the model’s reading of it. Keep review in your policy and keep the labels on your side.

Pricing

A grounded request bills input tokens at the model’s rate plus $1.00 per 1,000 searches, billed when the search returns, even with zero results and even if the decision then fails. In the recorded run, grounded requests averaged 1,906 input tokens: 1,906 × $0.02 per million = $0.000038 in tokens, plus $0.001 for the search, or $0.001038 per claim and $1.0381 per 1,000 claims. Ungrounded requests averaged 99 tokens, or $0.0020 per 1,000 claims. The search is 96% of the grounded cost: at the same input-token count, halving the token rate with the flash model (--model pplx-decider-flash-latest, $0.01 per million) saves about two cents per 1,000. No flash run was recorded for this page, so measure it on your claims before relying on that figure or its accuracy. A search that failed or timed out is not billed, and search_results does not say which searches ran, so the script prices every grounded attempt, retries included, and reports grounded attempts as an upper bound on billed searches. See Pricing.

Next steps

Your claims. Replace claims.jsonl. Put the claim first in the state, keep the question short, and run check on a sample: the warning and the claim text in the search query count tell you before any request whether each complete claim is in its query, and the URLs under each row tell you what the search found.
Several questions. Each question gets its own query, up to 100 questions and 10 searches at a time; questions with the same query share a search, and empty queries are skipped. Each search returns up to 5 results and the request keeps up to 10, or one per query when there are more than 10 queries, ordered by rank. The question part of each query is capped at 128 bytes when the state has text. Web grounding has the full rules, including criteria, choice and score questions, and object and array state.
Your cutoffs. Change SUPPORTED_MIN and CONTRADICTED_MAX, rerun eval, and watch all three bounds in the summary. The JSON in out/ has every probability and every URL, so you can replay cutoffs without new requests.
Read the docs. Web grounding covers the query, result, time, and failure rules in full; the Decisions API quickstart covers the question types.

​Why web grounding

​What you need

​Set up

​The claims

​Shape the request

​The query rule

​Constants and the padding helper

​One request per claim

​Verdicts and the scorecard

​Command line

​The full file

​Run it

​Measure it

​What the search sees

​Read the numbers

​Pricing

​Next steps