AI quality control

The dangerous thing about AI isn’t that it’s wrong.
It’s that it sounds right.

A language model sounds roughly equally competent whether it has read the source, remembers something similar, inferred something reasonable — or simply filled a gap. Linguistic confidence is not a quality signal. This page describes the four mechanical controls my factory uses to keep confident machines honest — and a catalogue of ten real failure patterns, each with the countermeasure that now prevents it. Not from a course. From production.

The principle

Move truth out of the wording, into the artifacts.

You cannot QA an agent by judging how convincing it sounds — it always sounds convincing. The entire defence rests on one move: truth-checking shifts from the agent’s formulation to observable artifacts — diffs, test runs, previews, screenshots, live platform readbacks. What follows is that principle applied four times.

Control 1

Prove you actually read it.

Early on, a specialist agent delivered convincing analyses of a dataset it had no access to. The text was well-structured, specific, professional. It just wasn’t an analysis of the source — it was a plausible reconstruction of what such an analysis would look like. That episode produced a ground rule:

An agent gets no credit for analysing a source until the system can show which source was opened, when, and which concrete observations the claim rests on.

Every strong operational claim is therefore classified by evidence strength:

  • Source-based — directly supported by readback or artifact.
  • Mixed — source plus professional interpretation.
  • Heuristic — qualified judgment without full readback.
  • Unverified — a claim without source evidence. Labelled as exactly that.

“The system says” is not evidence. “The agent analysed it” is not evidence either. And state must be fresh: an automated report once described a CI run as in progress when live readback showed it had already finished. The report wasn’t malicious — just stale. Live state beats the agent’s description of state. Always.

Control 2

The explanation is not the deliverable. The artifact is.

An agent cannot make its own answer true by writing “done”. Doneness demands different proof depending on the work:

Code change

  • The diff
  • Tests
  • CI result
  • Updated requirement status

User interface

  • Working preview
  • Screenshots at relevant breakpoints
  • Page / flow smoke test
  • Visual approval

Runtime / third-party action

  • API or platform readback
  • Timestamp
  • Concrete result
  • Error & rollback status

Analysis / architecture

  • Source map
  • Stated assumptions
  • Observation vs. judgment, separated
  • Open uncertainties & precise scope
Control 3

Adversarial review — mandate diversity, not just model diversity.

Friendly re-reading has a known weakness: the reviewer adopts the author’s framing. If a design says “this solution is secure”, a friendly reviewer checks whether the security argument is well written. An adversarial reviewer gets the opposite mandate: find the strongest way to break it. Find the hidden assumption. Find the path where a privileged actor bypasses the control. Find where the documentation promises more than the system enforces. Find the case where every test is green and the user outcome is still wrong.

A second model given the same task can confirm the same blindness. A reviewer with an explicit refutation mandate looks for something else entirely. In practice the chain looks like this:

  1. Architect designs the boundary
  2. Implementation agent builds
  3. QA reviewer reads the artifacts
  4. Adversarial reviewer attacks the premises
  5. The human makes the GO / HOLD call

No AI role is ever allowed to ratify its own work. The same actor may never define the requirement, build the solution, declare it correct and approve it into production — that’s segregation of duties, applied to AI.

Control 4

Uncertainty is a state — not a gap to fill with text.

When important information is missing, the correct system state is unverified, blocked, awaiting readback, quarantined or requires human decision — never a fluent paragraph that papers over the hole. In tightly bounded runtime work the rule is absolute:

Unknown = blocked. If scope, file classification, test expectation, rollback or authority is unclear, the agent stops. It may not “solve” the ambiguity by expanding its own mandate.
The failure catalogue

Ten real patterns — and the mechanics that stop them.

Every one of these was observed in live operation. Publishing them is the point: if your AI vendor can’t show you their failure catalogue, they haven’t run long enough to have one.

1 · Plausible analysis without source access

SymptomA detailed, professional analysis with few visible signs of uncertainty.
Root causeThe model lacks access, but can reconstruct what such an analysis probably looks like.
CountermeasureSource proof, claim-to-evidence mapping, explicit “unverified” status when readback is missing.

2 · Done-inflation

SymptomA change is called finished because the file exists, the PR is open, or the build is green.
Root causeImplementation, verification and production status collapsed into one comfortable word.
CountermeasureSeparate build/verify states, a ban on generic “done”, pre-done gate, delivery report, evidence requirements.

3 · Drift under pressure

SymptomCommits and text keep coming, but scope creeps, tests thin out, quality dies inside.
Root causeThe agent optimises for momentum and absorbs ambiguity instead of stopping.
CountermeasureSmall slices, max file scope, explicit forbidden moves, stop conditions, new dispatch required on scope growth.

4 · Too many hats

SymptomOne agent interprets the need, writes the architecture, builds, reviews, declares quality, proposes merge.
Root causeRole boundaries written as personalities instead of authority.
CountermeasureSeparate roles for architecture, orchestration, implementation, review and human approval. No self-ratification.

5 · Plausible synthesis without analysis

SymptomAn elegant summary of many documents that tests no contradictions and produces no new observations.
Root causeCoherent language mistaken for analytical work.
CountermeasureRequire findings, counter-evidence, source conflicts, concrete gaps — and observation, inference and recommendation kept separate.

6 · Stale-state hallucination

SymptomThe agent reports wrong PR, CI, deploy or runtime status.
Root causeStatus pulled from memory, an older report, or a different moment than the decision.
CountermeasureFresh readback on time-sensitive decisions. Git, platform and runtime state beat chat and documentation.

7 · Visually false green

SymptomTests are green, but the interface is empty, cropped, hidden, disabled — or missing the one thing that matters.
Root causeDOM, build and unit tests can’t judge composition, first paint or human comprehension.
CountermeasurePreview, screenshots, responsive checks, page smoke — and a separate visual approval before merge.

8 · Authority smuggling

SymptomA status ledger, a briefing or a helpful note quietly starts assigning roles or expanding mandates.
Root causeDescriptive artifacts gradually treated as normative ones.
CountermeasureAn authority hierarchy with explicit supersession — a ledger consumes authority; it does not create it.

9 · Hidden deviations

SymptomA rule was bypassed “temporarily”, but the work later parades as normally verified.
Root causeThe exception only ever existed in a conversation or a PR comment.
CountermeasureA deviation register, a distinct bypass status, a named owner and a visible normalisation task.

10 · Placeholder & internal leakage

SymptomTest text, internal codes, session references or credentials reach public output.
Root causeSource code and public build output treated as the same information zone.
CountermeasureSecret scans, internal-leak scans, placeholder greps — run against the final static output, not just the source.
The honest part

What the defence does not fully catch.

Strong, not magic. It can still miss:

A test that proves the wrong thing — and a screenshot that looks right while hiding bad interaction.
Several agents sharing the same blind spot — and a correct source that is itself outdated or wrong.
Semantic errors that syntax and CI checks don’t understand — and human approval that has gone routine.
A well-documented but fundamentally bad product idea. No gate catches that; judgment does.

Which is why human judgment remains part of the architecture. The goal was never to make AI infallible. The goal is to make errors visible, traceable, bounded, reversible — and much harder to promote to production.

Why this matters to you

Your AI initiative will meet every one of these patterns.

The question is whether it meets them with a defence already standing — or discovers them one production incident at a time. I’ve built the defence once. I can help you build yours.

Talk to me Inside the factory