Early in my factory’s life, a specialist agent delivered an analysis of a dataset. It was well-structured, specific, professionally worded. It cited patterns in the data. It drew reasonable conclusions.
It had never opened the dataset.
Not because it was broken. Because it couldn’t — it lacked access — and instead of saying so, it did what language models do brilliantly: it reconstructed what such an analysis would probably look like. The result was plausible enough that it survived two days of review before an instinct made me hand the same task to a second agent. My stomach was right. The analysis was fiction with excellent posture.
Here’s what makes this dangerous, and why I’d bet money your organisation’s AI pilot has the same problem: a language model sounds roughly equally competent in four very different situations. When it has read the source. When it remembers something similar. When it has inferred something reasonable. And when it has simply filled a gap. The tone is identical. The confidence is identical. The formatting is identical.
Linguistic confidence is not a quality signal. It’s a stylistic constant.
Your SteerCo has spent decades learning to detect a project manager who’s bluffing — the hedged phrasing, the eye contact, the suspiciously round numbers. None of those instincts work on an AI. It doesn’t hedge when it’s guessing. It doesn’t know it’s guessing.
Stop evaluating the claim. Demand the evidence.
The fix cannot be “read the output more carefully.” The fix is structural, and it’s the same one auditors have used for a century.
In my factory, every operational claim an agent makes is classified by evidence strength. Source-based: directly supported by an artifact or a live readback. Mixed: source plus interpretation. Heuristic: qualified judgment, no full readback. Unverified: a claim with nothing under it — labelled as exactly that, in writing. An agent gets no credit for “analysing” anything until the system can show which source was opened, when, and which observations the conclusion rests on.
“The system says” is not evidence. “The agent analysed it” is not evidence. A confident paragraph is not evidence. And this rule costs almost nothing to enforce — which is precisely why it’s embarrassing how few AI initiatives enforce it.
One question to ask tomorrow
If you have an AI pilot running right now, ask its owner: “When it gives us an answer, how do we know whether it read the source or reconstructed it?” If the answer is a pause, your pilot isn’t lying to you yet.
But it will. Structurally. And it will sound wonderful doing it.