AI hallucinations in customer-facing products: how product teams reduce and handle them

An AI hallucination is an output that sounds plausible but is wrong, such as an invented policy, a fake reference or an incorrect number. Product teams reduce hallucinations by narrowing what the system is asked to do, grounding its answers in approved sources, testing it on real and awkward cases, and refusing or escalating when the sources do not support an answer. Because hallucinations cannot be eliminated completely, the product must also decide what happens when one gets through: who sees it, how it is caught and how a customer is put right.

The UK government's AI Playbook describes hallucination, which it also calls confabulation, as large language models and other probabilistic generative models creating content that appears plausible but may be factually incorrect, because they generate the most likely output from patterns in their training data without an actual understanding of the content. UK government AI Playbook

Why do hallucinations matter more in financial products?

In a chat about recipes, a wrong answer is an annoyance. In a financial product it can be a wrong fee, a promised refund that policy does not allow, or an incorrect statement about a customer's account. Those errors create complaints, cost money and, in regulated services, can breach rules about clear and fair communication. The impact of a wrong answer should decide how much control you build, not the novelty of the feature.

How do product teams reduce hallucinations?

Five product decisions do most of the work:

1. Narrow the task. "Answer questions about our refund policy using the policy document" is far safer than "help customers with anything." Every capability you add is a new way to be wrong.
2. Ground answers in approved sources. Retrieve the relevant passages from current, approved documents and instruct the system to answer only from them, with a citation the user can check. Grounding merchant-support answers shows the pattern in practice.
3. Allow "I don't know". A system that must always answer will invent. Design an explicit refusal or hand-off when the sources do not cover the question.
4. Keep facts out of the model. Account balances, prices and limits should come from your systems of record, inserted into the answer, rather than generated.
5. Constrain the output. Structured formats, short answers and fixed wording for sensitive statements reduce room for invention.

How do you test for hallucinations?

Build an evaluation set from real questions, then add the cases most likely to fail:

Score each answer against an agreed definition of correct, set a release bar before testing, and rerun the set whenever the model, prompt or sources change. Evaluating AI features before release covers the method in more detail.

What should happen when a hallucination gets through?

Plan for it as you would any production incident:

• Let users flag bad answers in the interface, and route flags to a named owner. The AI Playbook also stresses having systems that let users report issues and prompt a human review.
• Log what was asked, what sources were retrieved and what was answered, so you can reproduce the failure.
• Put the customer right. Decide in advance how you honour, correct or apologize for a wrong statement, especially one about money.
• Fix the cause, not only the answer. Was a source missing, out of date or ambiguous? Was the task too broad?
• Keep a human review step where an error would affect money, eligibility or legal obligations. The Playbook requires human oversight to validate outputs in high-impact or high-risk situations.

Who owns hallucination risk?

The product manager owns the decision about what the system may say and do, the release bar and the response plan. Engineering owns retrieval quality and logging. Operations owns the review queue. Legal or compliance reviews fixed wording for sensitive statements. If nobody owns the flagged-answer queue, the controls exist only on paper. For choosing where to start with AI at all, see AI in operations: the first use case.

FAQ

What is an AI hallucination?

An AI output that sounds plausible but is factually wrong or unsupported, such as an invented policy, figure or citation.

Can AI hallucinations be eliminated?

Not entirely. They can be reduced substantially by narrowing the task, grounding answers in sources and testing, and handled through review and correction processes.

Does retrieval-augmented generation stop hallucinations?

It reduces them by giving the model relevant sources, but the model can still misread or go beyond those sources. Test grounded answers as carefully as any other.

How do you measure hallucination rate?

Run a fixed evaluation set of real and adversarial questions, mark each answer against an agreed definition of correct, and track the share of unsupported or wrong answers over time.

Should a customer-facing AI ever say "I don't know"?

Yes. A clear refusal with a route to a person is far better than a confident wrong answer.