In February 2024, a British Columbia tribunal ordered Air Canada to pay C$812.02 to a customer whose bereavement-fare question its website chatbot had answered wrongly — the bot described a refund policy that did not exist, while the airline's own site said otherwise. Air Canada argued the chatbot was a separate entity responsible for its own statements. The tribunal rejected that outright and held the airline liable for negligent misrepresentation.
The dollar amount is trivial. The principle is not: a business is bound by what its software tells a customer. Before you put an AI in front of your callers or your staff, it is worth understanding why these systems state false things with a straight face. The reason is structural. It is not a bug that next year's model will fix.
The machine predicts words. It does not look anything up.
A language model is trained on an enormous body of text to do one thing: given some words, produce the words most likely to come next. When you ask it a question, it is not consulting a database of facts. It is continuing your sentence in the most plausible way it can, drawing on patterns absorbed during training.
Most of the time, plausible and true point in the same direction. That is why the technology is useful at all. But the model carries no internal marker that separates "I saw this stated a thousand times" from "this merely resembles things I have seen." When the pattern runs out, the machine keeps writing anyway. There is no gauge that drops to empty.
The wrong answer sounds exactly like the right one
Ask a person about a shingle warranty they have never read and you will hear hesitation. Ask a model, and you get the same fluent, specific, well-organized paragraph you would get if it knew the document cold. Fluency is a property of the writing, not of the knowledge behind it.
This is what makes the failure expensive in a business. A wrong price delivered hesitantly gets double-checked. A wrong price delivered in a confident sentence, next to a plausible part number, gets quoted to a customer and honored at a loss — or argued about in front of a tribunal.
Retrieval narrows the problem. It does not close it.
The standard fix is retrieval: before the model answers, the system searches your actual documents — the price book, the warranty terms, the manufacturer bulletins — pulls the passages that bear on the question, and instructs the model to answer only from those passages and cite them. This is the pattern our private document search is built on, and it genuinely changes the odds.
It does not change them to zero. Researchers at Stanford tested the AI legal research tools sold by LexisNexis and Thomson Reuters — purpose-built retrieval systems marketed on the promise of eliminating fabrication. Their study, published in the Journal of Empirical Legal Studies in 2025, found one tool produced false statements on roughly 17 percent of test queries and the other on roughly 33 percent, against roughly 43 percent for a general model with no retrieval at all. The vendors disputed parts of the methodology. Grant every objection and the direction still holds: retrieval substantially reduces invention and does not eliminate it. Anyone who tells you otherwise is selling something.
The most dangerous question is the one your files never answer
When the answer genuinely is not in the documents, a well-built system says so. The model's default behavior is different: it assembles something plausible from whatever nearby text the search returned. Ask about a wind exclusion your warranty never mentions, and an unguarded system will construct one out of adjacent clauses, complete with a section reference.
Teaching models to refuse unanswerable questions is an active research area, and published work shows real improvement — without reaching zero. Refusal has to be designed in and then tested against questions you know have no answer. It cannot be assumed.
Sometimes the fabrication starts in your filing cabinet
A retrieval system is only as honest as what it retrieves, and business paperwork has failure modes of its own.
Text carries no expiration date. Your 2019 price sheet and your current one read almost identically to a search system, because they are about the same things. Nothing in the mathematics prefers the current one; that preference has to be built in, using effective dates attached when documents are loaded. Two versions of the same procedure in the index is worse — the system can blend them into instructions that were never true in either version, and a blend is harder to spot than a wrong pick.
Scanned paper adds another layer. A photographed, coffee-stained bulletin that reads "not permitted" can be digitized as "now permitted," and every component downstream will cite it faithfully, page number and all. And splitting documents into searchable pieces can sever a clause from the section that scopes it: a fragment ending in "this exclusion does not apply," retrieved alone, inverts the meaning of a policy. Exact identifiers fail in the opposite way — to a meaning-based search, Model 4400-B and Model 4400-C are nearly the same thing, which is why a serious build pairs it with old-fashioned exact matching.
What engineering cannot remove, and the habits that catch the rest
No configuration eliminates the residue, and the tools that claim to automatically detect fabricated answers are themselves unreliable — you cannot fully automate the checking of the automation. So the honest design accepts a residue and plans for it. Answers cite the page they came from, so a person can verify in seconds. The system is pointed at your own trained staff before it is ever pointed at customers, because an experienced tech can smell a wrong answer and a customer cannot. Anything customer-facing that touches money — prices, warranty scope, refund terms — goes through a person before it goes out, because of the principle the Air Canada case settled. And someone reads a sample of transcripts every week, the same way you would spot-check a new hire.
Where to start
Pick the ten questions your office answers most often and find the document that answers each one. If the document exists, is current, and carries a date, you have the raw material for a system that cites its sources. Where no such document exists, that gap — not the software — is the first thing worth fixing, and it costs nothing but an afternoon.