Check a Chart for Misleading Design with a Decision Model

A line chart with two value axes can make marketing spend look like it drives signups. Each axis is scaled on its own, so the two lines land on top of each other no matter what the data says. A bar chart whose axis starts at 93 makes a 5% difference look like a 5x difference. No number on either chart is wrong. The drawing is, which is why nothing in your data pipeline catches it: the query is right, the totals reconcile, and the chart still tells the wrong story to the people deciding next quarter’s budget. This recipe sends a chart image to pplx-decider-latest with yes/no questions about how the chart is drawn: does the bar axis start above zero, do two value axes make unrelated series look linked, do pie labels add to more than 100%, and six more. The same request asks three questions about a short brand guide: are the data colors on palette, is there a source line, is the title left-aligned. The decision model returns a probability for each question, plus a chart type and an overall score. Your Python decides which checks apply, compares the probabilities to cutoffs, labels the chart, and prints the reason for every finding:

charts/spend_signups__misleading.png
  design: misleading  (line_or_area 0.99, overall 1.45 of 3)
    0.99  Two value axes with mismatched scales
          Why: Each axis was scaled separately, so the two lines track each
          other by construction. The overlap suggests a link the data does not
          show.
  brand:  conforms

A live run of the finished script on four of the test charts. The left pane shows the chart being checked. The terminal first prints the twelve checks the script sends with every image, then runs the checker and prints every probability the decision model returned. The honest temperature chart at the end starts its axis at 60 and still scores clear, because the script does not apply the truncated-baseline check to line charts.

You can point the finished script at any PNG, JPEG, or WebP: a screenshot, an export from Tableau or Looker, or a figure from a notebook. The page also shows how to measure a checker like this one. A script draws every test chart twice from the same data, once honest and once with a single planted problem, and writes the answer key before any API call. You tune on a development set, freeze the checklist and cutoffs, and score a held-out set once. The held-out run below caught 3 of 3 design problems and 3 of 3 brand violations with no false flags, and the page reports the one experimental check that missed, and why.

Why a decision model

A chat model can look at the same chart and write a paragraph about it. You cannot put a cutoff on a paragraph, count how often it is right, or rerun it tomorrow and get the same answer. A decision model returns a calibrated probability for each question you ask, so your code can hold the policy: which checks apply to a bar chart, where FLAG sits, and what happens between clear and misleading. The policy is readable, testable, and yours to change without touching the model. Three things follow from that, and this recipe uses all three:

Numbers you can evaluate. Draw 41 charts with known problems, run them, and the result is a table: caught, passed, false flags. You know how good the checker is before you trust it.
One request per chart. Fourteen questions about one image go in one request, come back in about half a second, and cost about four thousandths of a cent in input tokens. Output tokens are free. That is a nightly sweep of every chart your company publishes, not a sample.
Reasons in your words. The model returns probabilities. Your checklist carries the sentence that explains each check, so the output says why a chart misleads in words you approved, not words a model generated.

What you need

Python 3.11 or newer on macOS or Linux. The shell commands use Unix syntax. On a Mac, check with python3 --version. The python3 that ships with Apple’s command line tools can be older; install a newer one with brew install python@3.12 or from python.org.
A Perplexity API key. Every key can call the Decisions API; there is no separate signup.

The Decisions API is one JSON endpoint, POST https://api.perplexity.ai/v1/decisions, so the script calls it with httpx, a plain HTTP client. Pillow resizes images. matplotlib draws the test charts in the second half of the page. Nothing else is needed.

Set up

Everything lives in one directory. Checking a chart takes three files: requirements.txt, checklist.py, and chart_check.py. Measuring the checker takes three more, which come later on the page.

decisions-api-chart-check/
├── requirements.txt       # four pinned dependencies
├── checklist.py           # the questions, the reasons, and the brand guide: edit this one
├── chart_check.py         # the request, the policy, and the CLI
├── make_charts.py         # draws the test charts and the answer key, no API calls
├── evaluate.py            # scores chart_check.py against the answer key
├── test_chart_check.py    # offline tests, no network
├── charts/                # written by make_charts.py
├── results.jsonl          # appended by chart_check.py and evaluate.py
└── .venv/                 # virtual environment from the install step

Start with the dependencies, pinned to the versions this recipe was tested with.

requirements.txt

httpx==0.28.1
matplotlib==3.11.2
pillow==13.0.0
pytest==9.1.1

Create the directory and save requirements.txt in it. Then create a virtual environment, install the packages, and export your key.

Install and set the key

mkdir decisions-api-chart-check && cd decisions-api-chart-check
# save requirements.txt here, then continue
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
export PERPLEXITY_API_KEY="your-api-key-here"

The checklist

checklist.py is plain data, and it is the file you edit to make this checker yours. CHART_TYPES becomes a choice question, so the response has one chart type and a confidence. Each entry in CHECKS becomes a noul (yes/no) question with written criteria for yes and for no, and the response has one probability per check. Each entry also carries a reason: one sentence, written by you, about what the problem does to a reader. The model never writes it; your code prints it next to the probability, so the output says why a chart misleads, in words you approved. APPLIES_TO lists the chart types each check makes sense for. OVERALL_RUBRIC becomes a score question whose four levels run from faithful to deceptive. EXPERIMENTAL names the checks that are measured but not trusted on their own; there is one, and the results section shows why. BRAND_GUIDE is a short brand guide in plain text. It goes into the request next to the image, and each entry in BRAND_CHECKS asks whether the chart breaks one of its rules. A brand entry carries a fix instead of a reason. Replace the three rules with your own, but keep each one to something visible in the image: colors, a source line, title placement, a logo. Write criteria the way you would brief a new analyst. “The first tick is not 0” and “for example 1990, 2000, 2010, 2015” give the model something concrete to look for. Cover the cases where the rule does not apply, too: the title rule says a chart with no title is fine, because without that sentence the model split the difference at about 0.6 on untitled charts. For every question, yes is the problem, so a high probability always means something to fix.

checklist.py

"""What chart_check.py asks about every chart. Edit this file to change the checks.

Each design check is (short label, why it misleads, what yes means, what no means).
Each brand rule is (short label, what to fix, what yes means, what no means).
For every question, yes is the problem.
"""

# Checks that are measured but not reliable enough to count in a headline number.
# parts_dont_sum asks the model to add numbers across labels, and it scores a
# 130% pie at 0.67 and a 123% donut at 0.04. Keep it visible, do not trust it alone.
EXPERIMENTAL = {"parts_dont_sum"}

CHART_TYPES = {
    "bar": "Vertical or horizontal bars or columns",
    "line_or_area": "Lines or filled areas over an ordered axis",
    "pie_or_donut": "Slices of a circle or ring",
    "scatter_or_bubble": "Points positioned by two values",
    "map": "Geographic map",
    "table_or_text": "Mostly numbers or text, not a plotted shape",
    "dashboard": "Several different charts shown together",
    "other": None,
}

CHECKS: dict[str, tuple[str, str, str, str]] = {
    "truncated_baseline": (
        "Bar axis does not start at zero",
        "Readers compare bar lengths. With the axis cut above zero, a small difference between bars looks several times larger than it is.",
        "A bar or column chart's value axis starts above zero (the first tick is not 0), so bar lengths exaggerate the differences between bars.",
        "The bars start at zero, or the chart has no bars.",
    ),
    "dual_axis_mismatch": (
        "Two value axes with mismatched scales",
        "Each axis was scaled separately, so the two lines track each other by construction. The overlap suggests a link the data does not show.",
        "Two series are plotted against two different value axes (one on the left, one on the right) with scales chosen so the series appear to move together, suggesting a relationship.",
        "There is one value axis, or the two axes share a scale or an index.",
    ),
    "inverted_axis": (
        "Value axis runs the wrong way",
        "Readers expect up to mean more. Flipped, a rising count looks like a fall.",
        "A value axis runs opposite to convention: larger values appear lower on the vertical axis or further left on the horizontal axis, so increases look like decreases.",
        "Values increase upward and to the right as usual.",
    ),
    "uneven_intervals": (
        "Irregular time intervals drawn as equal",
        "Ten-year and one-year gaps get the same width, so recent change looks steeper than earlier change.",
        "The horizontal axis shows time points that are unequally spaced in time (for example 1990, 2000, 2010, 2015, 2018, 2019) but drawn at equal distances, as if the gaps were the same.",
        "Time points are evenly spaced, or they are positioned in proportion to the actual gaps, or the axis is not time.",
    ),
    "parts_dont_sum": (
        "Pie or donut parts do not add to 100%",
        "A pie promises that the parts make a whole. Labels that add to more or less than 100% mean the slices cannot be trusted.",
        "A pie or donut chart's labeled percentages add up to clearly more or less than 100%, or slice sizes do not match their labels.",
        "The labeled parts add to about 100% and slice sizes match them, or the chart is not a pie or donut.",
    ),
    "title_overclaims": (
        "Title claims more than the data shows",
        "Most readers take the headline and skip the axis. The title promises a change the plotted values do not show.",
        "The title or a caption asserts a trend, surge, collapse, record or large difference that the plotted values do not support (for example calling a change of about one percentage point a surge).",
        "The title describes the plotted values fairly, or makes no claim about trend or size.",
    ),
    "missing_axis_labels": (
        "Value axis has no readable scale",
        "Without numbers on the axis, a reader cannot tell whether the differences are 1% or 100%.",
        "A value axis has no tick labels, so a reader cannot tell the magnitude or units of what is plotted.",
        "Value axes have tick labels, or the chart type needs none (a labeled pie chart, for example).",
    ),
    "perspective_3d": (
        "3D perspective distorts sizes",
        "Perspective makes nearer bars or slices look bigger and farther ones smaller, so the eye misreads the values.",
        "The chart is drawn in 3D or tilted perspective, so slices or bars nearer the viewer look larger than their values and farther ones look smaller.",
        "The chart is flat, with no perspective or depth effects.",
    ),
    "cumulative_disguise": (
        "Cumulative total hides a per-period decline",
        "A running total can only rise. Monthly numbers that are falling show up as a curve that still climbs.",
        "The chart plots a cumulative or running total that can only rise, while the subject (sales, downloads, users) is normally reported per period, so a slowdown or decline is hidden in a flattening curve.",
        "Values are shown per period, or a cumulative view is the natural reading of the subject.",
    ),
}

# A design check only counts on the chart types listed for it.
APPLIES_TO: dict[str, set[str]] = {
    "truncated_baseline": {"bar", "dashboard"},
    "dual_axis_mismatch": {"line_or_area", "bar", "dashboard"},
    "inverted_axis": {"line_or_area", "bar", "scatter_or_bubble", "dashboard"},
    "uneven_intervals": {"line_or_area", "bar", "scatter_or_bubble", "dashboard"},
    "parts_dont_sum": {"pie_or_donut", "dashboard"},
    "title_overclaims": set(CHART_TYPES),
    "missing_axis_labels": {"line_or_area", "bar", "scatter_or_bubble", "dashboard"},
    "perspective_3d": {"bar", "pie_or_donut", "line_or_area", "dashboard", "other"},
    "cumulative_disguise": {"line_or_area", "bar", "dashboard"},
}

OVERALL_RUBRIC = [
    "Faithful: the design does not distort what the data shows",
    "Minor: a presentation flaw that could confuse but is unlikely to mislead",
    "Misleading: the design makes differences or trends look much larger or smaller than the data supports",
    "Deceptive: the design pushes a conclusion the data does not support",
]

# Your brand guide, sent with every chart. Replace it with your own rules.
BRAND_GUIDE = (
    "Chart standards: "
    "1. Data colors: bars, lines, areas, and slices use only dark teal, light teal, orange, and rust. "
    "No blue, green, red, purple, or yellow. Text, axes, and backgrounds can be black, white, or gray. "
    "2. Source: the small note at the bottom left of every chart names the data source after the word 'Source:'. "
    "3. Title: the title is bold and left-aligned above the chart, not centered."
)

BRAND_CHECKS: dict[str, tuple[str, str, str, str]] = {
    "off_brand_colors": (
        "Data colors are outside the brand palette",
        "Redraw the data in dark teal, light teal, orange, or rust.",
        "At least one bar, line, area, or slice is a color the brand guide does not allow, such as blue, green, red, purple, or yellow.",
        "Every bar, line, area, and slice is dark teal, light teal, orange, or rust.",
    ),
    "missing_source": (
        "No source line",
        "Add 'Source:' and the data source to the note at the bottom left.",
        "No text anywhere at the bottom of the chart contains the word 'Source:' followed by where the data came from.",
        "Somewhere in the small text at the bottom of the chart, the word 'Source:' is followed by where the data came from, even if other text comes first.",
    ),
    "centered_title": (
        "Title is not left-aligned",
        "Align the title to the left edge of the chart.",
        "The chart title is centered or right-aligned above the chart.",
        "The chart title is left-aligned above the chart, or the chart has no title.",
    ),
}

The script, stage by stage

The rest of this section walks through chart_check.py one piece at a time. Every block below is an exact excerpt of the full file, which follows the excerpts.

Imports and constants

The endpoint, the model, and the four cutoffs live at the top. pplx-decider-latest selects the newest decision model. The cutoffs are the whole policy: a probability at or above FLAG means the problem is present, one between BORDERLINE and FLAG sends the chart to a person, and an overall score at or above REVIEW_SCORE does the same for design even when no single check fires. TYPE_CONFIDENCE sends a chart to review when the model is not sure what kind of chart it is, because the gating in judge depends on that answer.

Imports and constants

"""Check chart images for misleading design and brand standards with a decision model.

One Decisions API request per image returns a probability for each design
check and each brand rule in checklist.py, plus a chart type and an overall
score. The cutoffs and rules in this file turn those numbers into two labels:
clear, review, or misleading for design, and conforms, review, or off-brand
for the brand guide.

    python chart_check.py charts/revenue_bars__misleading.png
    python chart_check.py --no-brand deck/*.png      # design checks only
"""

from __future__ import annotations

import argparse
import base64
import email.utils
import io
import json
import os
import sys
import textwrap
import time
from dataclasses import asdict, dataclass, field

import httpx
from PIL import Image

from checklist import APPLIES_TO, BRAND_CHECKS, BRAND_GUIDE, CHART_TYPES, CHECKS, EXPERIMENTAL, OVERALL_RUBRIC

DECISIONS_URL = "https://api.perplexity.ai/v1/decisions"
MODEL = "pplx-decider-latest"

# Cutoffs. Freeze these before you score held-out charts.
FLAG = 0.60  # at or above: the problem is present
BORDERLINE = 0.40  # from here up to FLAG: a person should look
REVIEW_SCORE = 1.5  # an overall score (0 to 3) this high alone sends a chart to review
TYPE_CONFIDENCE = 0.5  # below this, the chart type is a guess and so is the gating: review

Fitting the image budget

The API reads images in 32 × 32 pixel tiles and accepts up to 2,048 tiles per image. An image over the budget does not return 400. The request waits about a minute and then returns 504 (Decisions API docs). So resizing is a correctness step. A full-screen screenshot on a Retina display is often 2880 × 1800 pixels, over 5,000 tiles, and would time out without it. fit_to_tiles rounds the width and height to the nearest multiple of 32, as the docs describe, and shrinks the image only when it is over budget. It flattens transparent images onto white first, because some exporters store black pixels under a transparent background, and a model would see a black chart. It also converts JPEG and WebP input to PNG, so the request always sends one format. The test charts are 900 × 560, about 500 tiles, and pass through unchanged.

fit_to_tiles()

def fit_to_tiles(image_bytes: bytes, max_tiles: int = 2048) -> bytes:
    """Return a PNG that fits the API's budget of 2,048 tiles of 32 x 32 pixels."""
    im = Image.open(io.BytesIO(image_bytes))
    if im.mode in ("RGBA", "LA", "P"):  # flatten transparency onto white, not black
        rgba = im.convert("RGBA")
        im = Image.alpha_composite(Image.new("RGBA", rgba.size, "white"), rgba)
    im = im.convert("RGB")
    w, h = im.size
    tiles = max(1, round(w / 32)) * max(1, round(h / 32))
    if tiles > max_tiles:
        scale = (max_tiles / tiles) ** 0.5 * 0.98  # a little margin for rounding
        im = im.resize((max(32, int(w * scale)), max(32, int(h * scale))), Image.LANCZOS)
    out = io.BytesIO()
    im.save(out, format="PNG")
    return out.getvalue()

One image, fourteen questions

build_request puts the image in state as a data URL, next to a one-line task and the brand guide. state accepts an object, so the image, the task, and the guide arrive as labeled fields. The API never fetches a URL; an http image URL returns 400. Pass context to add a sentence such as the dashboard name. Pass brand=False to leave the guide and its three questions out, which is what the --no-brand flag does for charts that were never meant to follow your guide. All fourteen questions go in one request: the chart type, nine design checks, three brand rules, and the overall score. They are facets of one review of one image, which is the case where the docs say grouping helps. Grouping also changes the numbers, because each question sees the others. Keep the checklist and its order fixed while you tune cutoffs; the results section has a real chart that moved from 0.61 to 0.45 when the three brand questions joined the request.

yes_no() and build_request()

def yes_no(yes: str, no: str) -> dict:
    return {"type": "noul", "instructions": "Judge only what is visible in the screenshot.", "criteria": {"true": yes, "false": no}}


def build_request(png: bytes, context: str | None = None, brand: bool = True) -> dict:
    """One image, fourteen questions: chart type, nine design checks, three brand rules, overall."""
    url = "data:image/png;base64," + base64.b64encode(png).decode()
    state: dict = {
        "task": "Review this chart screenshot for misleading design choices" + (" and for the brand guide." if brand else "."),
        "chart": {"type": "image_url", "image_url": {"url": url}},
    }
    if brand:
        state["brand_guide"] = BRAND_GUIDE
    if context:
        state["context"] = context
    questions: dict = {"chart_type": {"type": "choice", "instructions": "What kind of chart is shown?", "criteria": CHART_TYPES}}
    for name, (_, _, yes, no) in CHECKS.items():
        questions[name] = yes_no(yes, no)
    for name, (_, _, yes, no) in BRAND_CHECKS.items() if brand else ():
        questions[name] = yes_no(yes, no)
    questions["overall"] = {
        "type": "score",
        "instructions": "How likely is a typical reader to draw a wrong conclusion from this chart because of how it is designed?",
        "criteria": OVERALL_RUBRIC,
    }
    return {"model": MODEL, "state": state, "questions": questions}

Sending the request

decide posts the body with the client you pass in. On 429 it waits out Retry-After, which can be a number of seconds or an HTTP date; on a 5xx it backs off and retries, up to three times. Anything else that is not 200 raises with the status, the start of the error body, and the x-request-id header, which is what support asks for. By default an organization can send 10 requests per second; this script sends one at a time.

retry_after() and decide()

def retry_after(value: str | None) -> float:
    """Seconds to wait from a Retry-After header, which can be a number or an HTTP date."""
    if value is None:
        return 1.0
    try:
        return max(0.0, float(value))
    except ValueError:
        date = email.utils.parsedate_to_datetime(value)
        return max(0.0, date.timestamp() - time.time()) if date else 1.0


def decide(client: httpx.Client, body: dict, retries: int = 3) -> tuple[dict, httpx.Response]:
    """POST one request. Wait out 429s and retry 5xx with backoff."""
    for attempt in range(retries + 1):
        response = client.post(DECISIONS_URL, json=body)
        if attempt < retries and response.status_code == 429:
            time.sleep(retry_after(response.headers.get("Retry-After")))
        elif attempt < retries and response.status_code >= 500:
            time.sleep(2**attempt)
        else:
            break
    if response.status_code != 200:
        request_id = response.headers.get("x-request-id", "none")
        raise RuntimeError(f"{response.status_code} {response.text[:300]} (x-request-id {request_id})")
    return response.json(), response

From probabilities to two labels

The model answers every design check for every chart, including checks that make no sense for it. A line chart whose axis starts at 60 °F is fine; the same axis on a bar chart is not. judge counts a design check only when the chart type the model returned is listed for it in APPLIES_TO. A high probability on a check that does not apply goes to suppressed, where you can still see it. Brand rules apply to every chart. Then judge sorts the findings by probability and picks two labels. For design, any flag makes the chart misleading, and a borderline check, a high overall score, or an uncertain chart type makes it review. For brand, any flag makes it off-brand and a borderline rule makes it review; with --no-brand the brand label is skipped. Everything else is clear and conforms. The two labels are independent: a chart can be on-brand and misleading, or honest and off-brand. misleading still means a person should decide; it means the pattern is there with high probability, not that someone meant to deceive.

Verdict, label(), and judge()

@dataclass
class Verdict:
    label: str  # design: clear | review | misleading
    brand: str  # brand guide: conforms | review | off-brand, or skipped with --no-brand
    chart_type: str
    chart_confidence: float
    overall: float
    flagged: list[str] = field(default_factory=list)  # design checks at or above FLAG that apply
    borderline: list[str] = field(default_factory=list)  # design checks from BORDERLINE to FLAG that apply
    suppressed: list[str] = field(default_factory=list)  # high, but not for this chart type
    brand_flagged: list[str] = field(default_factory=list)  # brand rules at or above FLAG
    brand_borderline: list[str] = field(default_factory=list)
    probabilities: dict[str, float] = field(default_factory=dict)


def label(flagged: list, borderline: list, bad: str, good: str, extra_review: bool = False) -> str:
    return bad if flagged else "review" if borderline or extra_review else good


def judge(answers: dict) -> Verdict:
    """Application policy: gate design checks by chart type, then apply the cutoffs."""
    chart_type, confidence = answers["chart_type"]["choice"], float(answers["chart_type"]["confidence"])
    probs = {name: float(answers[name]["noul"]) for name in [*CHECKS, *BRAND_CHECKS] if name in answers}
    overall = float(answers["overall"]["score"])
    v = Verdict("", "", chart_type, confidence, overall, probabilities=probs)
    for name, p in sorted(probs.items(), key=lambda item: -item[1]):
        if name in BRAND_CHECKS:
            if p >= FLAG:
                v.brand_flagged.append(name)
            elif p >= BORDERLINE:
                v.brand_borderline.append(name)
        elif p >= FLAG:
            (v.flagged if chart_type in APPLIES_TO[name] else v.suppressed).append(name)
        elif p >= BORDERLINE and chart_type in APPLIES_TO[name]:
            v.borderline.append(name)
    # A guessed chart type means guessed gating, so a person looks even when nothing fired.
    v.label = label(v.flagged, v.borderline, "misleading", "clear", overall >= REVIEW_SCORE or confidence < TYPE_CONFIDENCE)
    v.brand = label(v.brand_flagged, v.brand_borderline, "off-brand", "conforms") if any(n in probs for n in BRAND_CHECKS) else "skipped"
    return v

The report and the command line

check_file reads an image, fits it, sends it, and returns a flat record with both labels, every probability, the input tokens, the time, and the request id. explain turns each finding into two parts: the probability with the check’s short label, then the reason or fix from checklist.py. print_record prints the design label with a Why: line per finding and the brand label with a Fix: line per broken rule. main reads the key from PERPLEXITY_API_KEY, opens one httpx.Client that sends it as a bearer token, checks each file in order, and appends one JSON line per image to results.jsonl. A failure stops the run with a one-line message and writes nothing.

check_file(), explain(), print_record(), and main()

def check_file(client: httpx.Client, path: str, brand: bool = True) -> dict:
    """Read, resize, and send one image. Return a flat record for results.jsonl."""
    with open(path, "rb") as handle:
        body = build_request(fit_to_tiles(handle.read()), brand=brand)
    started = time.perf_counter()
    data, response = decide(client, body)
    verdict = judge(data["answers"])
    return {
        "file": path,
        **asdict(verdict),
        "input_tokens": data["usage"]["input_tokens"],
        "seconds": round(time.perf_counter() - started, 2),
        "request_id": response.headers.get("x-request-id"),
        "model": MODEL,
        "checked_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
    }


def explain(names: list[str], table: dict, probs: dict, prefix: str) -> list[str]:
    """One line per finding: the probability, the label, and the reason or fix from checklist.py."""
    lines = []
    for name in names:
        short, reason = table[name][0], table[name][1]
        lines.append(f"    {probs[name]:.2f}  {short}" + ("  (experimental check)" if name in EXPERIMENTAL else ""))
        lines += textwrap.wrap(f"{prefix}{reason}", 78, initial_indent="          ", subsequent_indent="          ")
    return lines


def print_record(r: dict) -> None:
    p = r["probabilities"]
    lines = [r["file"], f"  design: {r['label']}  ({r['chart_type']} {r['chart_confidence']:.2f}, overall {r['overall']:.2f} of 3)"]
    lines += explain(r["flagged"] + r["borderline"], CHECKS, p, "Why: ")
    if r["brand"] != "skipped":
        lines += [f"  brand:  {r['brand']}"] + explain(r["brand_flagged"] + r["brand_borderline"], BRAND_CHECKS, p, "Fix: ")
    print("\n".join(lines))
    print(f"  {r['input_tokens']:,} input tokens, {r['seconds']:.2f} s, request {r['request_id']}")


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    parser.add_argument("files", nargs="+", help="PNG, JPEG, or WebP chart images")
    parser.add_argument("--out", default="results.jsonl", help="append one JSON line per image")
    parser.add_argument("--no-brand", action="store_true", help="skip the brand guide; design checks only")
    args = parser.parse_args(argv)

    api_key = os.environ.get("PERPLEXITY_API_KEY")
    if not api_key:
        print("Set PERPLEXITY_API_KEY first.", file=sys.stderr)
        return 2
    records = []
    with httpx.Client(headers={"Authorization": f"Bearer {api_key}"}, timeout=90.0) as client:
        for path in args.files:
            try:
                records.append(check_file(client, path, brand=not args.no_brand))
            except (RuntimeError, httpx.HTTPError) as error:
                print(f"{path}: Decisions API failed: {error}", file=sys.stderr)
                return 1
            print_record(records[-1])
    with open(args.out, "a", encoding="utf-8") as handle:
        handle.writelines(json.dumps(r) + "\n" for r in records)
    return 0


if __name__ == "__main__":
    sys.exit(main())

The full file

Save this as chart_check.py. It is the excerpts above in order, nothing else.

Full code - chart_check.py

chart_check.py

"""Check chart images for misleading design and brand standards with a decision model.

One Decisions API request per image returns a probability for each design
check and each brand rule in checklist.py, plus a chart type and an overall
score. The cutoffs and rules in this file turn those numbers into two labels:
clear, review, or misleading for design, and conforms, review, or off-brand
for the brand guide.

    python chart_check.py charts/revenue_bars__misleading.png
    python chart_check.py --no-brand deck/*.png      # design checks only
"""

from __future__ import annotations

import argparse
import base64
import email.utils
import io
import json
import os
import sys
import textwrap
import time
from dataclasses import asdict, dataclass, field

import httpx
from PIL import Image

from checklist import APPLIES_TO, BRAND_CHECKS, BRAND_GUIDE, CHART_TYPES, CHECKS, EXPERIMENTAL, OVERALL_RUBRIC

DECISIONS_URL = "https://api.perplexity.ai/v1/decisions"
MODEL = "pplx-decider-latest"

# Cutoffs. Freeze these before you score held-out charts.
FLAG = 0.60  # at or above: the problem is present
BORDERLINE = 0.40  # from here up to FLAG: a person should look
REVIEW_SCORE = 1.5  # an overall score (0 to 3) this high alone sends a chart to review
TYPE_CONFIDENCE = 0.5  # below this, the chart type is a guess and so is the gating: review


def fit_to_tiles(image_bytes: bytes, max_tiles: int = 2048) -> bytes:
    """Return a PNG that fits the API's budget of 2,048 tiles of 32 x 32 pixels."""
    im = Image.open(io.BytesIO(image_bytes))
    if im.mode in ("RGBA", "LA", "P"):  # flatten transparency onto white, not black
        rgba = im.convert("RGBA")
        im = Image.alpha_composite(Image.new("RGBA", rgba.size, "white"), rgba)
    im = im.convert("RGB")
    w, h = im.size
    tiles = max(1, round(w / 32)) * max(1, round(h / 32))
    if tiles > max_tiles:
        scale = (max_tiles / tiles) ** 0.5 * 0.98  # a little margin for rounding
        im = im.resize((max(32, int(w * scale)), max(32, int(h * scale))), Image.LANCZOS)
    out = io.BytesIO()
    im.save(out, format="PNG")
    return out.getvalue()


def yes_no(yes: str, no: str) -> dict:
    return {"type": "noul", "instructions": "Judge only what is visible in the screenshot.", "criteria": {"true": yes, "false": no}}


def build_request(png: bytes, context: str | None = None, brand: bool = True) -> dict:
    """One image, fourteen questions: chart type, nine design checks, three brand rules, overall."""
    url = "data:image/png;base64," + base64.b64encode(png).decode()
    state: dict = {
        "task": "Review this chart screenshot for misleading design choices" + (" and for the brand guide." if brand else "."),
        "chart": {"type": "image_url", "image_url": {"url": url}},
    }
    if brand:
        state["brand_guide"] = BRAND_GUIDE
    if context:
        state["context"] = context
    questions: dict = {"chart_type": {"type": "choice", "instructions": "What kind of chart is shown?", "criteria": CHART_TYPES}}
    for name, (_, _, yes, no) in CHECKS.items():
        questions[name] = yes_no(yes, no)
    for name, (_, _, yes, no) in BRAND_CHECKS.items() if brand else ():
        questions[name] = yes_no(yes, no)
    questions["overall"] = {
        "type": "score",
        "instructions": "How likely is a typical reader to draw a wrong conclusion from this chart because of how it is designed?",
        "criteria": OVERALL_RUBRIC,
    }
    return {"model": MODEL, "state": state, "questions": questions}


def retry_after(value: str | None) -> float:
    """Seconds to wait from a Retry-After header, which can be a number or an HTTP date."""
    if value is None:
        return 1.0
    try:
        return max(0.0, float(value))
    except ValueError:
        date = email.utils.parsedate_to_datetime(value)
        return max(0.0, date.timestamp() - time.time()) if date else 1.0


def decide(client: httpx.Client, body: dict, retries: int = 3) -> tuple[dict, httpx.Response]:
    """POST one request. Wait out 429s and retry 5xx with backoff."""
    for attempt in range(retries + 1):
        response = client.post(DECISIONS_URL, json=body)
        if attempt < retries and response.status_code == 429:
            time.sleep(retry_after(response.headers.get("Retry-After")))
        elif attempt < retries and response.status_code >= 500:
            time.sleep(2**attempt)
        else:
            break
    if response.status_code != 200:
        request_id = response.headers.get("x-request-id", "none")
        raise RuntimeError(f"{response.status_code} {response.text[:300]} (x-request-id {request_id})")
    return response.json(), response


@dataclass
class Verdict:
    label: str  # design: clear | review | misleading
    brand: str  # brand guide: conforms | review | off-brand, or skipped with --no-brand
    chart_type: str
    chart_confidence: float
    overall: float
    flagged: list[str] = field(default_factory=list)  # design checks at or above FLAG that apply
    borderline: list[str] = field(default_factory=list)  # design checks from BORDERLINE to FLAG that apply
    suppressed: list[str] = field(default_factory=list)  # high, but not for this chart type
    brand_flagged: list[str] = field(default_factory=list)  # brand rules at or above FLAG
    brand_borderline: list[str] = field(default_factory=list)
    probabilities: dict[str, float] = field(default_factory=dict)


def label(flagged: list, borderline: list, bad: str, good: str, extra_review: bool = False) -> str:
    return bad if flagged else "review" if borderline or extra_review else good


def judge(answers: dict) -> Verdict:
    """Application policy: gate design checks by chart type, then apply the cutoffs."""
    chart_type, confidence = answers["chart_type"]["choice"], float(answers["chart_type"]["confidence"])
    probs = {name: float(answers[name]["noul"]) for name in [*CHECKS, *BRAND_CHECKS] if name in answers}
    overall = float(answers["overall"]["score"])
    v = Verdict("", "", chart_type, confidence, overall, probabilities=probs)
    for name, p in sorted(probs.items(), key=lambda item: -item[1]):
        if name in BRAND_CHECKS:
            if p >= FLAG:
                v.brand_flagged.append(name)
            elif p >= BORDERLINE:
                v.brand_borderline.append(name)
        elif p >= FLAG:
            (v.flagged if chart_type in APPLIES_TO[name] else v.suppressed).append(name)
        elif p >= BORDERLINE and chart_type in APPLIES_TO[name]:
            v.borderline.append(name)
    # A guessed chart type means guessed gating, so a person looks even when nothing fired.
    v.label = label(v.flagged, v.borderline, "misleading", "clear", overall >= REVIEW_SCORE or confidence < TYPE_CONFIDENCE)
    v.brand = label(v.brand_flagged, v.brand_borderline, "off-brand", "conforms") if any(n in probs for n in BRAND_CHECKS) else "skipped"
    return v


def check_file(client: httpx.Client, path: str, brand: bool = True) -> dict:
    """Read, resize, and send one image. Return a flat record for results.jsonl."""
    with open(path, "rb") as handle:
        body = build_request(fit_to_tiles(handle.read()), brand=brand)
    started = time.perf_counter()
    data, response = decide(client, body)
    verdict = judge(data["answers"])
    return {
        "file": path,
        **asdict(verdict),
        "input_tokens": data["usage"]["input_tokens"],
        "seconds": round(time.perf_counter() - started, 2),
        "request_id": response.headers.get("x-request-id"),
        "model": MODEL,
        "checked_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
    }


def explain(names: list[str], table: dict, probs: dict, prefix: str) -> list[str]:
    """One line per finding: the probability, the label, and the reason or fix from checklist.py."""
    lines = []
    for name in names:
        short, reason = table[name][0], table[name][1]
        lines.append(f"    {probs[name]:.2f}  {short}" + ("  (experimental check)" if name in EXPERIMENTAL else ""))
        lines += textwrap.wrap(f"{prefix}{reason}", 78, initial_indent="          ", subsequent_indent="          ")
    return lines


def print_record(r: dict) -> None:
    p = r["probabilities"]
    lines = [r["file"], f"  design: {r['label']}  ({r['chart_type']} {r['chart_confidence']:.2f}, overall {r['overall']:.2f} of 3)"]
    lines += explain(r["flagged"] + r["borderline"], CHECKS, p, "Why: ")
    if r["brand"] != "skipped":
        lines += [f"  brand:  {r['brand']}"] + explain(r["brand_flagged"] + r["brand_borderline"], BRAND_CHECKS, p, "Fix: ")
    print("\n".join(lines))
    print(f"  {r['input_tokens']:,} input tokens, {r['seconds']:.2f} s, request {r['request_id']}")


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    parser.add_argument("files", nargs="+", help="PNG, JPEG, or WebP chart images")
    parser.add_argument("--out", default="results.jsonl", help="append one JSON line per image")
    parser.add_argument("--no-brand", action="store_true", help="skip the brand guide; design checks only")
    args = parser.parse_args(argv)

    api_key = os.environ.get("PERPLEXITY_API_KEY")
    if not api_key:
        print("Set PERPLEXITY_API_KEY first.", file=sys.stderr)
        return 2
    records = []
    with httpx.Client(headers={"Authorization": f"Bearer {api_key}"}, timeout=90.0) as client:
        for path in args.files:
            try:
                records.append(check_file(client, path, brand=not args.no_brand))
            except (RuntimeError, httpx.HTTPError) as error:
                print(f"{path}: Decisions API failed: {error}", file=sys.stderr)
                return 1
            print_record(records[-1])
    with open(args.out, "a", encoding="utf-8") as handle:
        handle.writelines(json.dumps(r) + "\n" for r in records)
    return 0


if __name__ == "__main__":
    sys.exit(main())

Check your own chart

With the virtual environment active and the key exported, point the script at any chart image. Use --no-brand for a chart that was never meant to follow your brand guide, and pass several files to check them in one run.

Check a chart

python chart_check.py --no-brand ~/Desktop/chart.png

To see it on a chart you did not draw, save the truncated bar graph from Wikimedia Commons as a PNG (the file page has a PNG preview at several widths) and check it. Output observed on 2026-10-19 against pplx-decider-latest. The image checked here was a 1783 × 1114 pixel screenshot of the preview; a smaller download costs fewer tokens and can move the probabilities a little:

Observed output

truncated_bars.png
  design: misleading  (bar 0.98, overall 1.55 of 3)
    0.98  Bar axis does not start at zero
          Why: Readers compare bar lengths. With the axis cut above zero, a
          small difference between bars looks several times larger than it is.
  2,571 input tokens, 1.00 s, request 5945f866-cf90-4ae1-ae8c-02eccaf82e70

The script printed that from this response. The image is a bar chart, the bar axis is cut, and the other eight design checks came back near zero. overall is a score question, so it returns a position on the 0 to 3 rubric with a probability for each level.

The response behind it, abbreviated

{
  "model": "pplx-decider-latest",
  "answers": {
    "chart_type": {
      "type": "choice",
      "choice": "bar",
      "confidence": 0.983,
      "probabilities": {"bar": 0.985, "line_or_area": 0.009, "pie_or_donut": 0.003, ...}
    },
    "truncated_baseline": {"type": "noul", "noul": 0.983},
    "dual_axis_mismatch": {"type": "noul", "noul": 0.001},
    "inverted_axis":      {"type": "noul", "noul": 0.000},
    ... six more design checks, each under 0.002 ...
    "overall": {
      "type": "score",
      "score": 1.546,
      "confidence": 0.456,
      "legend": {"0": "Faithful: ...", "1": "Minor: ...", "2": "Misleading: ...", "3": "Deceptive: ..."},
      "probabilities": {"0": 0.147, "1": 0.204, "2": 0.603, "3": 0.045}
    }
  },
  "usage": {"input_tokens": 2571, "output_tokens": 11}
}

Everything from the first line of output to the last is a decision your code made from those numbers: that truncated_baseline applies to a bar chart, that 0.98 is over FLAG, that the label is misleading, and which sentence to print under it.

Measure it

The rest of the page is about knowing how good the checker is. That takes charts with known problems.

The test charts

make_charts.py draws 41 PNG files from 22 cases and writes charts/labels.json, the answer key. Each case is an honest chart plus the few settings its planted version changes, so you can read exactly what was planted: revenue_bars changes only ylim, csat_title changes only title, plan_colors changes only colors. The data, the chart kind, the labels, and the source line stay the same, so the only way to tell the two versions apart is the drawing. Every chart follows the brand guide unless its case plants a brand violation, so the misleading charts are on-brand and the off-brand charts are honest. That keeps the two kinds of check from leaning on each other. The cases fall into three sets:

Development (12 pairs): one pair for each of the nine design checks and each of the three brand rules. Tune on these.
Controls (3 honest charts): a temperature line chart with a 60 to 100 °F axis, a labeled log-scale chart, and a chart with no title. All three are fine designs that a careless checker flags.
Held out (7 pairs): dark theme, horizontal bars, a donut, an area chart, a pie. Score these once, after everything is frozen.

The answer key is written before any chart is drawn, and no file name, title, or note in an image says which version it is. The API sees pixels only. Run it once; it makes no network calls.

make_charts.py

"""Draw the test charts: an honest version and a planted-problem version of the same data.

Design cases write charts/<case>__honest.png and charts/<case>__misleading.png.
Brand cases write charts/<case>__honest.png and charts/<case>__off_brand.png.
charts/labels.json records the planted problem for every image. Nothing here
calls the API, and nothing in an image says which version it is.
"""

from __future__ import annotations

import json
from pathlib import Path

import matplotlib

matplotlib.use("Agg")  # draw to files, no window
import matplotlib.pyplot as plt

OUT = Path("charts")
BRAND = ["#20808D", "#8FD1D8", "#E6772E", "#A84B2F"]  # dark teal, light teal, orange, rust
OFF_BRAND = ["#1f77b4", "#d62728", "#2ca02c", "#9467bd"]  # matplotlib defaults: blue, red, green, purple
MONTHS = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"]
YEARS = ["1990", "2000", "2010", "2015", "2018", "2019", "2020"]
SPEND = [40, 42, 41, 45, 44, 47, 46, 48, 50, 49, 52, 51]
SIGNUPS = [1200, 1180, 1250, 1230, 1300, 1290, 1350, 1340, 1380, 1400, 1420, 1410]
INCIDENTS = [120, 135, 150, 170, 185, 210, 235, 260, 290, 310]
MONTHLY = [50, 46, 41, 37, 33, 29, 26, 23, 20]
CUMULATIVE = [sum(MONTHLY[: i + 1]) for i in range(len(MONTHLY))]
WEEKS = [f"W{i}" for i in range(1, 9)]
UNITS = [310, 340, 365, 390, 420, 410, 450, 470]


def index(xs: list[float]) -> list[float]:
    """Rescale a series so its first value is 100."""
    return [round(v / xs[0] * 100, 1) for v in xs]


# Each case is the honest chart plus the few keys the misleading version changes.
# Controls have no misleading version. "set" decides when a case is scored.
CASES = [
    {"id": "revenue_bars", "set": "dev", "pattern": "truncated_baseline",
     "honest": {"kind": "bar", "title": "Revenue by quarter, FY2026", "note": "Net revenue, $ millions. Source: Finance data mart.",
                "x": ["Q1", "Q2", "Q3", "Q4"], "y": [94.1, 95.8, 96.9, 98.7], "ylabel": "$ millions"},
     "misleading": {"ylim": (93, 99.5)}},
    {"id": "spend_signups", "set": "dev", "pattern": "dual_axis_mismatch",
     "honest": {"kind": "line", "title": "Marketing spend and new signups, 2026", "note": "Monthly. Source: Growth dashboard.",
                "x": MONTHS, "y": index(SPEND), "label": "Marketing spend (index)",
                "y_extra": index(SIGNUPS), "label_extra": "New signups (index)", "ylabel": "Index (Jan = 100)", "ylim": (90, 135)},
     "misleading": {"y": SPEND, "label": "Marketing spend ($K)", "ylabel": "Marketing spend ($K)", "ylim": (38, 54),
                    "y_extra": None, "y2": SIGNUPS, "label2": "New signups", "ylim2": (1150, 1450)}},
    {"id": "incidents_inverted", "set": "dev", "pattern": "inverted_axis",
     "honest": {"kind": "area", "title": "Security incidents per year", "note": "Confirmed incidents, 2015 to 2024. Source: Security operations.",
                "x": [str(y) for y in range(2015, 2025)], "y": INCIDENTS, "ylabel": "Incidents"},
     "misleading": {"invert": True}},
    {"id": "uneven_years", "set": "dev", "pattern": "uneven_intervals",
     "honest": {"kind": "line", "title": "Average home price", "note": "Metro area, $ thousands. Source: County assessor records.",
                "x": [int(y) for y in YEARS], "y": [120, 165, 240, 280, 330, 345, 390], "ylabel": "$ thousands", "xlabel": "Year"},
     "misleading": {"x": YEARS}},  # strings: matplotlib spaces categories evenly
    {"id": "market_share_pie", "set": "dev", "pattern": "parts_dont_sum",
     "honest": {"kind": "pie", "title": "Market share, 2026", "note": "Share of units sold. Source: Industry panel data.",
                "x": ["Our brand", "Competitor A", "Competitor B"], "y": [45, 35, 20]},
     "misleading": {"y": [52, 41, 37]}},  # labels add to 130%
    {"id": "csat_title", "set": "dev", "pattern": "title_overclaims",
     "honest": {"kind": "line", "title": "Customer satisfaction steady at 81 to 82%", "note": "CSAT score, monthly, n = 4,200 per month. Source: Support survey.",
                "x": MONTHS[:6], "y": [81.2, 81.5, 81.1, 81.9, 81.7, 82.0], "ylabel": "CSAT (%)", "ylim": (0, 100)},
     "misleading": {"title": "Customer satisfaction surges to a record high"}},
    {"id": "unlabeled_axis", "set": "dev", "pattern": "missing_axis_labels",
     "honest": {"kind": "bar", "title": "Churn by plan", "note": "Trailing 90 days. Source: Billing system.",
                "x": ["Free", "Starter", "Team", "Enterprise"], "y": [8.4, 5.1, 3.2, 1.1], "ylabel": "Churn (%)"},
     "misleading": {"hide_ticks": True, "ylabel": ""}},
    {"id": "cumulative_downloads", "set": "dev", "pattern": "cumulative_disguise",
     "honest": {"kind": "area", "title": "App downloads, 2026", "note": "Monthly. Source: App store console.",
                "x": MONTHS[:9], "y": MONTHLY, "ylabel": "Downloads per month (thousands)"},
     "misleading": {"y": CUMULATIVE, "ylabel": "Cumulative downloads (thousands)"}},
    {"id": "regional_3d", "set": "dev", "pattern": "perspective_3d",
     "honest": {"kind": "bar", "title": "Sales by region, Q3 2026", "note": "Share of unit sales, percent. Source: Sales ops.",
                "x": ["North", "South", "East", "West"], "y": [31, 27, 24, 18], "ylabel": "Share of sales (%)"},
     "misleading": {"kind": "bar3d"}},
    {"id": "plan_colors", "set": "dev", "brand": "off_brand_colors",
     "honest": {"kind": "bar", "title": "Active seats by plan", "note": "October 2026, thousands. Source: Billing system.",
                "x": ["Free", "Starter", "Team", "Enterprise"], "y": [42, 18, 11, 6], "ylabel": "Seats (thousands)"},
     "off_brand": {"colors": OFF_BRAND}},
    {"id": "latency_source", "set": "dev", "brand": "missing_source",
     "honest": {"kind": "line", "title": "API p95 latency by week", "note": "Milliseconds. Source: Gateway metrics.",
                "x": WEEKS, "y": [212, 205, 198, 201, 190, 186, 181, 179], "ylabel": "p95 latency (ms)"},
     "off_brand": {"note": ""}},
    {"id": "nps_title", "set": "dev", "brand": "centered_title",
     "honest": {"kind": "bar", "title": "Net Promoter Score by region", "note": "Q3 2026. Source: Customer survey.",
                "x": ["Americas", "EMEA", "APAC"], "y": [41, 37, 44], "ylabel": "NPS"},
     "off_brand": {"center_title": True}},
    {"id": "temp_line_nonzero", "set": "control", "pattern": None,
     "honest": {"kind": "line", "title": "Average daily temperature, Phoenix, first week of October", "note": "Degrees Fahrenheit. Source: NOAA station PHX.",
                "x": [f"Oct {d}" for d in range(1, 8)], "y": [88, 87, 85, 86, 84, 83, 85], "ylabel": "°F", "ylim": (60, 100)}},
    {"id": "no_title", "set": "control", "pattern": None,
     "honest": {"kind": "bar", "title": "", "note": "Open tickets by priority, today. Source: Help desk.",
                "x": ["P0", "P1", "P2", "P3"], "y": [3, 18, 44, 61], "ylabel": "Open tickets"}},
    {"id": "log_scale_labeled", "set": "control", "pattern": None,
     "honest": {"kind": "line", "title": "Requests per day, 2021 to 2026 (log scale)", "note": "Daily API requests, logarithmic axis. Source: Gateway metrics.",
                "x": [str(y) for y in range(2021, 2027)], "y": [12e3, 48e3, 210e3, 1.1e6, 5.2e6, 24e6],
                "ylabel": "Requests per day (log scale)", "log": True}},
    {"id": "h_margin_bars", "set": "holdout", "pattern": "truncated_baseline", "dark": True,
     "honest": {"kind": "hbar", "title": "Gross margin by product line", "note": "FY2026, percent. Source: Finance.",
                "x": ["Hardware", "Software", "Services", "Support"], "y": [62, 71, 66, 74], "ylabel": "Gross margin (%)", "ylim": (0, 100)},
     "misleading": {"ylim": (60, 75)}},
    {"id": "h_dual_climate", "set": "holdout", "pattern": "dual_axis_mismatch", "dark": True,
     "honest": {"kind": "line", "title": "Store temperature and ice cream units sold", "x": WEEKS, "y": UNITS, "label": "Units sold", "ylabel": "Units sold",
                "note": "Weekly, summer 2026. Store temperature ranged 72 to 79 °F over the same weeks. Source: Store sensors, POS."},
     "misleading": {"y": [72, 73, 74, 75, 77, 76, 78, 79], "label": "Store temperature (°F)", "ylabel": "Temperature (°F)",
                    "ylim": (71, 80), "y2": UNITS, "label2": "Units sold", "ylim2": (300, 480), "note": "Weekly, summer 2026. Source: Store sensors, POS."}},
    {"id": "h_pie_sum", "set": "holdout", "pattern": "parts_dont_sum", "dark": True,
     "honest": {"kind": "donut", "title": "How teams spend their week", "note": "Share of reported hours. Source: Internal time survey.",
                "x": ["Meetings", "Focus work", "Email and chat", "Other"], "y": [28, 42, 18, 12]},
     "misleading": {"y": [38, 42, 28, 15]}},  # labels add to 123%
    {"id": "h_title_claim", "set": "holdout", "pattern": "title_overclaims", "dark": True,
     "honest": {"kind": "bar", "title": "Reported thefts per 1,000 residents, roughly unchanged", "note": "By year. Source: City police annual report.",
                "x": ["2022", "2023", "2024", "2025"], "y": [4.1, 4.0, 4.2, 4.3], "ylabel": "Per 1,000 residents", "ylim": (0, 6)},
     "misleading": {"title": "Theft rate explodes across the city"}},
    {"id": "h_signup_colors", "set": "holdout", "brand": "off_brand_colors",
     "honest": {"kind": "area", "title": "Weekly signups", "note": "Thousands. Source: Growth dashboard.",
                "x": WEEKS, "y": [8.2, 8.9, 9.4, 9.1, 10.3, 10.8, 11.6, 12.0], "ylabel": "Signups (thousands)"},
     "off_brand": {"colors": ["#9467bd"]}},
    {"id": "h_share_title", "set": "holdout", "brand": "centered_title", "dark": True,
     "honest": {"kind": "pie", "title": "Support tickets by channel", "note": "September 2026. Source: Help desk.",
                "x": ["Email", "Chat", "Phone"], "y": [48, 37, 15]},
     "off_brand": {"center_title": True}},
    {"id": "h_cost_source", "set": "holdout", "brand": "missing_source", "dark": True,
     "honest": {"kind": "hbar", "title": "Cloud spend by team", "note": "September 2026, $ thousands. Source: Cloud billing export.",
                "x": ["Search", "Ads", "Infra", "Data"], "y": [182, 96, 240, 131], "ylabel": "$ thousands"},
     "off_brand": {"note": ""}},
]


def draw(spec: dict, dark: bool, path: Path) -> None:
    """Render one chart spec to a 900 x 560 PNG."""
    plt.style.use("dark_background" if dark else "default")
    kind, x, y, colors = spec["kind"], spec["x"], spec["y"], spec.get("colors", BRAND)
    fig = plt.figure(figsize=(9, 5.6), dpi=100)
    ax = fig.add_subplot(projection="3d" if kind == "bar3d" else None)
    if kind == "bar":
        ax.bar(x, y, color=colors if "colors" in spec else colors[0])
    elif kind == "hbar":
        ax.barh(x, y, color=colors[0])
    elif kind in ("line", "area"):
        ax.plot(x, y, marker="o", lw=3, color=colors[0], label=spec.get("label"))
        if kind == "area":
            ax.fill_between(range(len(y)) if isinstance(x[0], str) else x, y, alpha=0.3, color=colors[0])
        if spec.get("y_extra"):
            ax.plot(x, spec["y_extra"], marker="o", lw=3, color=colors[1], label=spec["label_extra"])
            ax.legend(loc="upper left")
    elif kind in ("pie", "donut"):
        ring = {"width": 0.45} if kind == "donut" else None
        ax.pie(y, labels=[f"{name}\n{v}%" for name, v in zip(x, y)], colors=colors,
               startangle=90, counterclock=False, wedgeprops=ring, textprops={"fontsize": 13})
    elif kind == "bar3d":
        ax.bar3d(range(len(y)), 0, 0, 0.6, 0.6, y, color=colors[0], shade=True)
        ax.set_xticks([i + 0.3 for i in range(len(x))], x)
        ax.set_yticks([])
        ax.set_zlabel(spec["ylabel"])
        ax.view_init(elev=28, azim=-58)
    if spec.get("y2"):  # a second value axis on the right
        ax2 = ax.twinx()
        ax2.plot(x, spec["y2"], marker="s", lw=3, color=colors[1], label=spec["label2"])
        ax2.set_ylim(*spec["ylim2"])
        ax2.set_ylabel(spec["label2"])
        ax.legend(loc="upper left")
        ax2.legend(loc="lower right")
    value_lim = ax.set_xlim if kind == "hbar" else ax.set_ylim
    value_label = ax.set_xlabel if kind == "hbar" else ax.set_ylabel
    if kind not in ("pie", "donut", "bar3d"):
        if spec.get("log"):
            ax.set_yscale("log")
        else:
            value_lim(*spec.get("ylim", (0, None)))
        value_label(spec.get("ylabel", ""))
        if spec.get("xlabel"):
            ax.set_xlabel(spec["xlabel"])
        if spec.get("invert"):
            ax.invert_yaxis()
        if spec.get("hide_ticks"):
            ax.tick_params(axis="y", left=False, labelleft=False)
        ax.spines[["top", "right"]].set_visible(bool(spec.get("y2")))
    if spec.get("center_title"):
        fig.suptitle(spec["title"], fontsize=16, fontweight="bold")
    else:
        fig.suptitle(spec["title"], fontsize=16, fontweight="bold", x=0.02, ha="left")
    fig.text(0.02, 0.015, spec["note"], fontsize=10, alpha=0.75)
    fig.savefig(path)
    plt.close(fig)


def main() -> None:
    OUT.mkdir(exist_ok=True)
    labels = []
    for case in CASES:
        for variant in ("honest", "misleading", "off_brand"):
            if variant == "honest" or variant in case:
                labels.append({"file": f"{OUT}/{case['id']}__{variant}.png", "case": case["id"], "set": case["set"],
                               "variant": variant,
                               "pattern": case.get("pattern") if variant == "misleading" else None,
                               "brand": case.get("brand") if variant == "off_brand" else None})
    # The answer key is written before any chart exists, and long before any API call.
    (OUT / "labels.json").write_text(json.dumps(labels, indent=2))
    for case in CASES:
        for label in (row for row in labels if row["case"] == case["id"]):
            spec = {**case["honest"], **case.get(label["variant"], {})}
            draw(spec, case.get("dark", False), Path(label["file"]))
    print(f"Wrote {len(labels)} charts and {OUT}/labels.json")


if __name__ == "__main__":
    main()

Draw the charts, then check two of them

python make_charts.py
python chart_check.py charts/spend_signups__misleading.png charts/spend_signups__honest.png charts/plan_colors__off_brand.png

Observed output

charts/spend_signups__misleading.png
  design: misleading  (line_or_area 0.99, overall 1.45 of 3)
    0.99  Two value axes with mismatched scales
          Why: Each axis was scaled separately, so the two lines track each
          other by construction. The overlap suggests a link the data does not
          show.
  brand:  conforms
  1,842 input tokens, 0.89 s, request 46c483c4-00d5-4a1b-8045-e55c5b4b6809
charts/spend_signups__honest.png
  design: clear  (line_or_area 1.00, overall 0.06 of 3)
  brand:  conforms
  1,842 input tokens, 1.99 s, request dfaca84f-e9f2-42d3-bb6c-6d150e43a3c1
charts/plan_colors__off_brand.png
  design: clear  (bar 1.00, overall 0.05 of 3)
  brand:  off-brand
    1.00  Data colors are outside the brand palette
          Fix: Redraw the data in dark teal, light teal, orange, or rust.
  1,842 input tokens, 0.46 s, request abb5a6ed-c55c-43d8-a843-dc9f02cf5198

Scoring

evaluate.py runs chart_check.py over a set from charts/labels.json and counts, separately for design and for brand: planted problems caught (the planted check is in flagged or brand_flagged), other charts passed (clear for design, conforms for brand), and false flags (any flag on a chart where nothing of that kind was planted). Checks listed in EXPERIMENTAL get their own row so they cannot inflate or sink the design number. It also prints the score of a checker that sends every chart to review. That checker looks careful and does nothing: it catches no problem and passes no chart. Then it lists every miss.

evaluate.py

"""Score chart_check.py against the answer key in charts/labels.json.

    python evaluate.py --set dev,control   # tune cutoffs and wording on these
    python evaluate.py --set holdout       # once, after everything is frozen
"""

from __future__ import annotations

import argparse
import json
import os
import sys

import httpx

import chart_check
from checklist import EXPERIMENTAL

PRICE_PER_M_INPUT = 0.02  # USD per million input tokens on pplx-decider-latest; output is free
EMPTY = {"planted": 0, "caught": 0, "fine": 0, "passed": 0, "false_flags": 0}


def tally(labels: list[dict], records: dict[str, dict]) -> tuple[dict, list[str]]:
    """Count catches, passes, and false flags per set for design and brand, and describe every miss.

    Planted experimental checks are counted on their own row, not in the design row.
    """
    table: dict[str, dict] = {}
    misses = []
    for lab in labels:
        r = records[lab["file"]]
        rows = table.setdefault(lab["set"], {"design": dict(EMPTY), "brand": dict(EMPTY)})
        for kind, planted, flagged, good in (("design", lab["pattern"], r["flagged"], "clear"),
                                             ("brand", lab["brand"], r["brand_flagged"], "conforms")):
            row, got = rows[kind], r["label"] if kind == "design" else r["brand"]
            if planted in EXPERIMENTAL:
                row = rows.setdefault("exper.", dict(EMPTY))
            if planted:
                row["planted"] += 1
                row["caught"] += planted in flagged
                if planted not in flagged:
                    misses.append(f"{lab['file']}: {planted} {r['probabilities'][planted]:.2f}, {kind} {got}")
            else:
                row["fine"] += 1
                row["passed"] += got == good
                row["false_flags"] += len(flagged)
                if got != good:
                    top = ", ".join(f"{n} {r['probabilities'][n]:.2f}" for n in flagged) or f"overall {r['overall']:.2f}"
                    misses.append(f"{lab['file']}: expected {kind} {good}, got {got} ({top})")
    return table, misses


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    parser.add_argument("--set", default="dev,control", help="comma-separated: dev, control, holdout")
    parser.add_argument("--out", default="results.jsonl", help="append one JSON line per image")
    args = parser.parse_args()

    api_key = os.environ.get("PERPLEXITY_API_KEY")
    if not api_key:
        print("Set PERPLEXITY_API_KEY first.", file=sys.stderr)
        return 2
    sets = args.set.split(",")
    with open("charts/labels.json", encoding="utf-8") as handle:
        labels = [lab for lab in json.load(handle) if lab["set"] in sets]

    records: dict[str, dict] = {}
    with httpx.Client(headers={"Authorization": f"Bearer {api_key}"}, timeout=90.0) as client:
        for lab in labels:
            try:
                records[lab["file"]] = chart_check.check_file(client, lab["file"])
            except (RuntimeError, httpx.HTTPError) as error:
                print(f"{lab['file']}: Decisions API failed: {error}", file=sys.stderr)
                return 1
            r = records[lab["file"]]
            print(f"{r['label']:<11}{r['brand']:<10}{lab['file']:<44}{r['input_tokens']:>6,} tokens {r['seconds']:>5.2f} s")
    with open(args.out, "a", encoding="utf-8") as handle:
        handle.writelines(json.dumps(r) + "\n" for r in records.values())

    table, misses = tally(labels, records)
    print(f"\n{'set':<9}{'check':<8}{'planted caught':>16}{'others passed':>15}{'false flags':>13}")
    for name, rows in table.items():
        for kind, row in rows.items():
            caught = f"{row['caught']} of {row['planted']}" if row["planted"] else "n/a"
            print(f"{name:<9}{kind:<8}{caught:>16}{row['passed']:>10} of {row['fine']:<3}{row['false_flags']:>12}")
    planted = sum(row["planted"] for rows in table.values() for row in rows.values())
    fine = sum(row["fine"] for rows in table.values() for row in rows.values())
    # The bar to beat: a checker that sends every chart to review looks careful and does nothing.
    print(f"always-review baseline: 0 of {planted} planted caught, 0 of {fine} others passed")
    for miss in misses:
        print("miss:", miss)
    tokens = sum(r["input_tokens"] for r in records.values())
    print(f"\n{len(records)} charts, {tokens:,} input tokens, ${tokens / 1e6 * PRICE_PER_M_INPUT:.5f}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Tune on the development set and the controls. Change the criteria or the cutoffs, rerun, repeat. When you are done, freeze checklist.py and chart_check.py, then score the held-out set once.

Score the development set, then the held-out set

python evaluate.py --set dev,control
python evaluate.py --set holdout

Results

Measured on 2026-10-19 against pplx-decider-latest, with the files on this page. Two changes came from the development set and the controls, and the cutoffs did not change. The first version of the source rule asked for a line that starts with “Source:”. The test charts put the source at the end of a note, and the model gave that rule 0.66 to 0.94 on 8 of 23 conforming charts, a fair reading of the rule as written. The rule now asks for “Source:” anywhere in the note. The first version of the title rule had no answer for a chart with no title, and the model gave it about 0.6 on untitled charts; the rule now says an untitled chart is fine, and the no-title control scores 0.04. After those two edits the files were frozen and the held-out set was scored once. The design checks and cutoffs come from an earlier version of this checker that was tested on different renderings of the four held-out design cases. Nothing in the design checks was changed in response to them. The three held-out brand cases are new.

Set	Check	Planted problem caught	Other charts passed
Development (12 pairs)	Design	8 of 8 (planted check 0.81 to 1.00)	15 of 15 (highest check 0.18)
	Brand	3 of 3 (each 1.00)	21 of 21 (highest rule 0.05)
	Experimental	1 of 1 (0.65)	n/a
Controls (3 honest charts)	Design	n/a	3 of 3 (every check 0.04 or lower)
	Brand	n/a	3 of 3
Held out (7 pairs)	Design	3 of 3	10 of 10
	Brand	3 of 3 (each 1.00)	11 of 11
	Experimental	0 of 1 (0.02)	n/a
Always-review baseline, all sets	Both	0 of 19	0 of 63

Observed output, held-out run

clear      conforms  charts/h_margin_bars__honest.png             1,842 tokens  0.70 s
misleading conforms  charts/h_margin_bars__misleading.png         1,842 tokens  0.42 s
clear      conforms  charts/h_dual_climate__honest.png            1,842 tokens  1.48 s
misleading conforms  charts/h_dual_climate__misleading.png        1,842 tokens  0.48 s
clear      conforms  charts/h_pie_sum__honest.png                 1,842 tokens  0.44 s
clear      conforms  charts/h_pie_sum__misleading.png             1,842 tokens  0.43 s
clear      conforms  charts/h_title_claim__honest.png             1,842 tokens  0.43 s
misleading conforms  charts/h_title_claim__misleading.png         1,842 tokens  0.50 s
clear      conforms  charts/h_signup_colors__honest.png           1,842 tokens  0.43 s
clear      off-brand charts/h_signup_colors__off_brand.png        1,842 tokens  0.45 s
clear      conforms  charts/h_share_title__honest.png             1,842 tokens  0.43 s
clear      off-brand charts/h_share_title__off_brand.png          1,842 tokens  0.43 s
clear      conforms  charts/h_cost_source__honest.png             1,842 tokens  0.43 s
clear      off-brand charts/h_cost_source__off_brand.png          1,842 tokens  1.82 s

set      check     planted caught  others passed  false flags
holdout  design            3 of 3        10 of 10            0
holdout  brand             3 of 3        11 of 11            0
holdout  exper.            0 of 1         0 of 0             0
always-review baseline: 0 of 7 planted caught, 0 of 21 others passed
miss: charts/h_pie_sum__misleading.png: parts_dont_sum 0.02, design clear

14 charts, 25,788 input tokens, $0.00052

The held-out request ids, in the order above, are 5353438d-fcf5-4f6e-8734-b85d9046360e, b54231cf-364c-4810-8ed5-4a33649ff178, b0b8f70d-8837-48c3-86f0-6b8a2e3ef9b8, a5f76ad8-bfe6-4c87-b243-2c4bb61adfc1, d23e9405-223a-49fa-b3b9-0d7a7849db6b, bba66963-686b-43e6-8823-143bf9a10c08, e93c58e5-798b-4059-a923-df2a1f7b961a, bc8cb4f2-63fa-4846-8a11-7c60c93c0795, 80ecf435-9364-4ab8-8499-846c3b6d4eb0, d7138a6f-979e-4c68-b2e7-15980545b016, db49cf9a-3b21-4167-89d1-041516554b71, 6097cc2c-a3e3-43d1-92c5-4d2be0d31a76, 48d1178c-2b17-4297-9863-b38ebeeab2cf, 44188ec6-beac-490e-be1c-11e9f0aaeb22. results.jsonl has every request id, probability, and token count from your own runs. The experimental check. parts_dont_sum asks whether pie labels add to more than 100%. The development pie, whose three labels add to 130%, scored 0.65, just over FLAG. The held-out donut, whose four labels add to 123%, scored 0.02 and was labeled clear. Adding numbers across labels is the weakest check on the list by a wide margin, which is why it is in EXPERIMENTAL and has its own row. Keep it visible, do not act on it alone, and if pie totals matter to you, check the source data in code. Run to run. Two development runs on 2026-10-19 with the final files returned identical probabilities for all 27 charts. Do not count on that: the Decisions API docs say pplx-decider-latest can move to a newer decision model, and you need to revalidate the cutoffs when it does. Your own charts drawn with the same script can also render slightly differently on another machine, which can move a probability near a cutoff. Cost and speed. Every test chart was 1,842 input tokens. At $0.02 per million input tokens, with output tokens free, scoring all 41 charts once costs about $0.0015, and a chart costs about $0.00004. Across the 41 requests in the development and held-out runs, the median was 0.45 seconds and 9 in 10 finished within 0.54 seconds. The three slowest took 1.0, 1.5, and 1.8 seconds, so plan for an occasional slow request.

Charts nobody drew for this page

The 41 test charts come from one script, so they show the checks work on clean, single-problem charts. They do not show what happens on charts drawn by other tools. After the held-out run, the frozen files were run with --no-brand on 11 screenshots from public sites: four sample dashboards from Tableau Public, a life expectancy chart from Our World in Data, five examples from the Wikipedia article on misleading graphs (the truncated bar graph and its full-scale version, a 3D pie and its flat version, and a two-bar chart with a cut axis), and a Federal Reserve chart of military spending with two value axes. None had been seen by this version of the checker before.

Every honest chart was clear except the Tableau stocks dashboard, which went to review because the model gave its chart type a confidence of 0.42. Dashboards mix chart kinds, and that rule is doing what it is for.
The truncated bar graph was misleading at 0.98, its full-scale twin clear at 0.00. The 3D pie was misleading at 1.00, the flat pie clear. The two-bar chart was misleading at 0.95.
The military spending chart, the kind of two-axis chart people argue about, came back at 0.61 for dual_axis_mismatch with 11 questions and at 0.45 with the brand guide and its three questions in the same request. Same image, same model, misleading one way and review the other. That is the grouping effect from the request section, measured, and it is why you freeze the checklist before you trust a cutoff and rescore everything when you add a question.

The no-title fix to the brand rule came from these screenshots, so they are not a clean held-out set for that rule. They are a clean set for the design checks: 7 of 7 honest charts passed (6 clear, 1 review), 4 of 4 misleading charts caught, 0 false flags. Label a sample of your own charts and run evaluate.py on them before you trust a cutoff on your dashboards.

Run it on a schedule

The practical shape of this recipe is a sweep, not a one-off. Export every published dashboard as an image each night, run chart_check.py over the folder, and send each misleading or review result to the chart’s owner with its Why: lines. The dashboards above cost about 2,500 input tokens each, so a chart costs about $0.00005; at the default 10 requests per second, that is 36,000 charts an hour for under $2. A GitHub Action that checks every PNG a notebook produces on a pull request is the same loop with a different trigger. results.jsonl already has everything a report needs: both labels, every probability, and a request id per chart.

Run the tests

The tests cover the request shape with and without the brand guide, the tile budget, transparent images, chart-type gating, the review rules including an uncertain chart type, brand and design labels staying separate, the printed reasons, the 429 retry with both forms of Retry-After, and the scoring with an experimental check. They make no network calls and need no key.

test_chart_check.py

"""Offline checks for chart_check.py and evaluate.py. No network, no API key.

Run: python -m pytest -q
"""

from __future__ import annotations

import io

import httpx
from PIL import Image

import chart_check as cc
from checklist import BRAND_CHECKS, CHECKS
from evaluate import tally


def png(w: int, h: int) -> bytes:
    buf = io.BytesIO()
    Image.new("RGB", (w, h), "white").save(buf, format="PNG")
    return buf.getvalue()


def answers(chart_type: str = "bar", overall: float = 0.2, **probs: float) -> dict:
    """A response's `answers` in the documented shape. Values are test inputs, not model output."""
    a = {"chart_type": {"type": "choice", "choice": chart_type, "confidence": 0.9},
         "overall": {"type": "score", "score": overall}}
    for name in [*CHECKS, *BRAND_CHECKS]:
        a[name] = {"type": "noul", "noul": probs.get(name, 0.01)}
    return a


def test_request_has_chart_type_design_checks_brand_rules_and_overall():
    body = cc.build_request(png(64, 64))
    assert body["model"] == "pplx-decider-latest"
    assert list(body["questions"]) == ["chart_type", *CHECKS, *BRAND_CHECKS, "overall"]
    assert body["state"]["chart"]["image_url"]["url"].startswith("data:image/png;base64,")
    assert "brand_guide" in body["state"]
    without = cc.build_request(png(64, 64), brand=False)
    assert list(without["questions"]) == ["chart_type", *CHECKS, "overall"] and "brand_guide" not in without["state"]


def test_large_images_shrink_to_the_tile_budget_and_small_ones_do_not():
    w, h = Image.open(io.BytesIO(cc.fit_to_tiles(png(3000, 2000)))).size
    assert round(w / 32) * round(h / 32) <= 2048
    assert Image.open(io.BytesIO(cc.fit_to_tiles(png(900, 560)))).size == (900, 560)


def test_transparent_background_becomes_white_not_black():
    buf = io.BytesIO()
    Image.new("RGBA", (64, 64), (0, 0, 0, 0)).save(buf, format="PNG")  # black under a transparent alpha
    sent = Image.open(io.BytesIO(cc.fit_to_tiles(buf.getvalue())))
    assert sent.mode == "RGB" and sent.getpixel((5, 5)) == (255, 255, 255)


def test_a_design_check_counts_only_on_the_chart_type_it_applies_to():
    bars = cc.judge(answers("bar", truncated_baseline=0.95))
    assert bars.label == "misleading" and bars.flagged == ["truncated_baseline"]
    line = cc.judge(answers("line_or_area", truncated_baseline=0.95))
    assert line.label == "clear" and line.suppressed == ["truncated_baseline"]


def test_borderline_probability_or_high_overall_score_means_review():
    assert cc.judge(answers("bar", title_overclaims=0.5)).label == "review"
    assert cc.judge(answers("pie_or_donut", overall=1.8)).label == "review"
    assert cc.judge(answers("pie_or_donut", overall=0.3)).label == "clear"


def test_brand_is_judged_separately_from_design_and_can_be_skipped():
    v = cc.judge(answers("bar", off_brand_colors=0.9))
    assert (v.label, v.brand, v.brand_flagged) == ("clear", "off-brand", ["off_brand_colors"])
    v = cc.judge(answers("bar", truncated_baseline=0.9, missing_source=0.5))
    assert (v.label, v.brand) == ("misleading", "review")
    design_only = {k: v for k, v in answers("bar").items() if k not in BRAND_CHECKS}
    assert cc.judge(design_only).brand == "skipped"


def test_a_guessed_chart_type_sends_the_chart_to_review():
    a = answers("bar")
    a["chart_type"]["confidence"] = 0.3
    assert cc.judge(a).label == "review"


def test_each_finding_prints_its_probability_then_its_reason():
    name, (short, reason, _, _) = next(iter(CHECKS.items()))
    lines = cc.explain([name], CHECKS, {name: 0.97}, "Why: ")
    assert lines[0].strip() == f"0.97  {short}" or lines[0].strip().startswith(f"0.97  {short}")
    assert " ".join(line.strip() for line in lines[1:]) == f"Why: {reason}"


def test_decide_waits_out_a_429_then_returns_the_answer(monkeypatch):
    replies = iter([httpx.Response(429, headers={"Retry-After": "0"}),
                    httpx.Response(200, json={"answers": answers(), "usage": {"input_tokens": 1}})])
    monkeypatch.setattr(cc.time, "sleep", lambda s: None)
    client = httpx.Client(transport=httpx.MockTransport(lambda request: next(replies)))
    data, response = cc.decide(client, {"model": cc.MODEL})
    assert response.status_code == 200 and data["usage"]["input_tokens"] == 1


def test_retry_after_accepts_seconds_or_an_http_date():
    assert cc.retry_after("2") == 2.0 and cc.retry_after(None) == 1.0
    assert cc.retry_after("Wed, 21 Oct 2015 07:28:00 GMT") == 0.0  # a past date means no wait


def test_tally_counts_a_missed_pattern_and_a_false_flag():
    labels = [{"file": "a", "set": "dev", "pattern": "parts_dont_sum", "brand": None},
              {"file": "b", "set": "dev", "pattern": None, "brand": None}]
    records = {"a": vars(cc.judge(answers("pie_or_donut", parts_dont_sum=0.1))),
               "b": vars(cc.judge(answers("bar", truncated_baseline=0.9)))}
    table, misses = tally(labels, records)
    assert table["dev"]["exper."] == {"planted": 1, "caught": 0, "fine": 0, "passed": 0, "false_flags": 0}
    assert table["dev"]["design"] == {"planted": 0, "caught": 0, "fine": 1, "passed": 0, "false_flags": 1}
    assert table["dev"]["brand"]["passed"] == 2 and len(misses) == 2

Run the tests

python -m pytest -q

What this is not

It does not check the data. A chart can start at zero, label every axis, and still plot the wrong numbers.
It cannot see what was left out. A chart that starts the date range at a convenient low point looks fine as an image. That is a question about the data behind the chart, not the pixels.
It does not judge intent. A bar chart of body temperature with an axis starting at 95 °F is a defensible choice. misleading means a pattern is present; a person decides what to do about it.
The reasons are yours, not the model’s. Each Why: and Fix: line is the sentence you wrote in checklist.py for that check. It explains the check, not this particular chart.
It does not check rendering. Clipped text, overlapping cards, a button pushed off screen, or a blank page are defects in how a page was drawn, and they need different questions than the ones here. This recipe assumes the chart rendered as its author intended and asks whether that rendering tells the truth.
It does not replace a brand review. Three rules about colors, a source line, and a title cover what is easy to see in an image. Fonts, logo use, and tone of voice need a person. The color rule catches blue where teal belongs, not a teal two shades off; an exact palette check is a pixel count you can do in Pillow without a model.

Adapt it

Your brand guide. Replace BRAND_GUIDE and BRAND_CHECKS with your rules, add an honest/off-brand pair for each to make_charts.py, and rerun the development set. Describe colors by name as well as by look (“dark teal, not blue”), because the model reads the image, not your hex codes. Say what happens when a rule does not apply, as the title rule does for untitled charts.
Your design checks. Add an entry to CHECKS and to APPLIES_TO, add a pair to make_charts.py, and rerun. Adding a question changes the numbers for the others, so rescore everything.
Your cutoffs. Label a few hundred of your own charts, run evaluate.py on them, and move FLAG and BORDERLINE. Leave margin around where your charts cluster; a cutoff on top of a cluster turns small movement into a different label.
Context. Pass context to build_request, for example “Quarterly board deck, finance team”, to give the model the setting the chart appears in.
The other decision model. Set MODEL = "pplx-decider-flash-latest" to try the smaller model. The docs recommend one question per request for Flash by default, and this recipe groups fourteen, so measure it on your own labeled charts first. The results on this page are for pplx-decider-latest only.

Troubleshooting

Every Decisions API error prints as <file>: Decisions API failed: <status> <error body> (x-request-id <id>).

Set PERPLEXITY_API_KEY first. means the key is not exported in this terminal. Run the export line again.
400 with Invalid model means MODEL is not a name the API accepts. Use pplx-decider-latest.
401 is a missing or invalid key.
504 after about a minute usually means an image over the tile budget. fit_to_tiles prevents that for files sent through check_file; if you build requests another way, resize first.
ModuleNotFoundError: No module named 'checklist' means checklist.py is not in the directory you run from. Save it next to chart_check.py, and run both chart_check.py and evaluate.py from that directory.
No matching distribution found for matplotlib==3.11.2 during install means the virtual environment uses Python older than 3.11. Remove .venv and create it again with a newer Python, for example python3.12 -m venv .venv.

​Why a decision model

​What you need

​Set up

​The checklist

​The script, stage by stage

​Imports and constants

​Fitting the image budget

​One image, fourteen questions

​Sending the request

​From probabilities to two labels

​The report and the command line

​The full file

​Check your own chart

​Measure it

​The test charts

​Scoring

​Results

​Charts nobody drew for this page

​Run it on a schedule

​Run the tests

​What this is not

​Adapt it

​Troubleshooting

Why a decision model

What you need

Set up

The checklist

The script, stage by stage

Imports and constants

Fitting the image budget

One image, fourteen questions

Sending the request

From probabilities to two labels

The report and the command line

The full file

Check your own chart

Measure it

The test charts

Scoring

Results

Charts nobody drew for this page

Run it on a schedule

Run the tests

What this is not

Adapt it

Troubleshooting