AI Sounds Confident. That Doesn't Make It Right.

A purchase, a click, or an unanswered email tells you something. The trouble starts when AI supplies the rest of the story, and the team stops asking questions.

Someone buys an expensive coffee machine from your website. The AI-generated profile describes them as a coffee enthusiast who values quality and will respond to premium accessories.

Perhaps. They might also have bought a wedding present. Or replaced the machine in the office kitchen. Or simply chosen the only model that could arrive before Friday. The transaction is real. The explanation needs checking.

This is not just a marketing problem. The same pattern shows up when AI summarises exit interviews into a tidy theme, turns a support-ticket backlog into a root cause, or compares two campaign reports and hands the board a recommendation. AI can help people find patterns and suggest what might explain them. It can also turn a thin piece of evidence into a detailed, confident story. That matters once the story starts shaping decisions.

Three Places Confidence Outruns the Evidence

1. A Behaviour Is Not a Motive

A purchase tells you what someone bought, when, and at what price. It may tell you very little about why. Consider three people booking the same Queenstown hotel: one celebrating an anniversary, one attending a conference, one needing somewhere to stay after a cancelled flight. Same hotel, different reasons entirely. If the AI describes all three as experience-seeking leisure travellers, the resulting campaign misses two of them.

A click carries the same ambiguity. Someone repeatedly visiting a delivery page might be struggling to find an answer, not showing intent to buy. Label them "high intent" and marketing sends another sales message when a clearer delivery explanation would have helped more.

What to do: ask recent customers what was happening that led them to look for this, and what made them choose you. Use AI to organise the answers once you have collected them, then check its themes against what people actually said.

2. An Absence Is Not a Single Explanation

A customer has not purchased in six months. The system flags them as a churn risk. AI drafts a win-back offer. Before it sends, consider what they bought: six months is a long absence for someone who regularly orders coffee beans, and hardly surprising for someone who bought a fridge.

Silence can reflect a changed need, a long buying cycle, or a perfectly satisfactory purchase. It can also mean dissatisfaction. The absence alone does not establish which. The same logic applies to an employee who has gone quiet after a strong start, or a client account with no recent activity: AI will happily supply a cause, but the absence of data is not evidence for any one of them.

What to do: check the normal rhythm for the product, the account, or the relationship before accepting the flagged explanation, and where appropriate, ask directly rather than acting on an inference.

3. A Generated Persona Is Not a Person

You ask an AI to act as a time-poor customer and review a proposal. It produces thoughtful objections, suggests better wording, and explains what would persuade it. That can help you prepare questions and explore possibilities. But the answers were generated. No customer saw the proposal or described their actual circumstances, and adding a name, a photograph, and a detailed biography does not change where the evidence came from.

What to do: label generated reactions as hypotheses. Choose the ones that would most affect your decision and investigate them with real people. If the AI says an offer feels confusing, ask people to explain it in their own words. (Research reference: Yoon and colleagues, NAACL 2024, on differences between simulated and human responses in conversational recommendation.)

Seven Checks Before You Act on an AI Report

This is where a second, related problem shows up: not the story AI tells about a person, but the recommendation it makes from a comparison. Say you upload two campaign reports and ask where to put next month's budget. Campaign B generated more revenue, so the AI recommends increasing its spend.

Sounds sensible. But Campaign B also ran for longer, received more budget, and targeted existing customers with a discount. Campaign A targeted people who had never bought from you. You cannot answer "which performed better" from revenue alone, and neither can the AI. A clear explanation can make an incomplete comparison feel settled. Before acting on an AI analysis of any performance data, run these seven checks. This example assumes the data and tool were already approved for the task; the question here is whether the analysis itself deserves your confidence.

Run those checks on the campaign example and a narrower, more honest finding emerges: Campaign B produced more recorded revenue, Campaign A produced more revenue per dollar spent, and their audiences and offers differ enough that the budget recommendation needs further evidence. That is a useful result. It tells the team what it knows and what to investigate next, which is usually a controlled test rather than an immediate budget shift.

The Three-Question Test

Try this in your next planning meeting. Choose one sentence from an AI-generated brief or report, something like "Our customers value convenience over price," and ask three questions:

• What did we observe? Perhaps customers selected the fastest delivery option.
• What have we inferred? That convenience generally matters more than price, which the observation alone does not establish.
• What would help us check it? A handful of customer conversations, a look at choices under different delivery prices, or a small test.

Bring what you learn back into the brief, and keep the uncertainty visible where it remains. Before approving a description of a customer, a market, or a result, ask one more question: which parts came from evidence, and which parts did the AI supply?