How to Audit an AI

Five questions that turn a confident answer into something you can inspect.

Compelle · Methods

Most AI systems ask you to trust an answer. An audit asks what would justify that trust. The useful starting point is not the fluency of the output but the evidence, instructions and evaluation behind it.

This guide draws on lessons from Compelle's historical adversarial-reasoning research. It describes a portable review process, not commands for a live API or a promise that every historical research record is publicly available.

1. What was it told to do?

Ask for the relevant instructions, the question as posed, the source material supplied and the model version. Distinguish the user's request from any system-level constraints. Missing instructions limit what you can conclude about why the model behaved as it did.

Check the framing before evaluating the answer. “Is this true for most people?” and “Is this true on average?” can produce different arguments. A precise answer to the wrong question is still the wrong deliverable.

2. Is the output consistent with those instructions?

Read the output against the stated task. Did it address the requested population and time period? Did it distinguish facts from assumptions? Did it actually consider the strongest opposing case?

Do not substitute a cosmetic check for a substantive one. Following a formatting rule is not evidence that a model followed an evidentiary standard. Record the specific gap between instruction and behavior.

3. Are its claims real?

Select the claims on which the conclusion depends and check them against primary sources. Open the cited document. Confirm that the quoted passage exists and supports the claim in context. Check dates, denominators and whether a comparison uses compatible populations.

When an answer includes a precise number without a source, treat that precision as something to investigate, not a reason to trust it. A debate transcript records what was said; it does not verify the speaker's assertions.

4. Does the evaluation answer the right question?

Inspect the judging criteria and, where available, the reasoning behind individual verdicts. Persuasiveness, factual correctness, instruction-following and usefulness for a decision are different targets. A score is only meaningful in relation to its target.

Separate results by relevant conditions: model version, prompt, topic, side assignment and evaluation period. An aggregate across changing conditions can conceal a reversal. Report disagreements and uncertainty rather than collapsing everything into a single confident winner.

5. Can someone else inspect the same evidence?

Preserve the inputs, outputs, evaluation criteria and versions used for the analysis. A changing service response is not a fixed research artifact. For a file-based evaluation, record the exact file version and a cryptographic hash so another reviewer can identify the same bytes.

State access limits plainly. If a source is private or no longer public, say so rather than implying that a reader can independently reproduce the analysis. A curated example illustrates a method; it does not reproduce the full research population.

What to do with the answers

Trust should attach to the claims that survived inspection, not to an entire system by default. An unsupported citation calls for source review. A framing error calls for a better question. A disagreement among judges calls for an explanation, not a hidden average.

Compelle's method, research and sample work show how we approach structured disagreement and evidence review. A custom engagement begins with your question and the standard of evidence it requires.