Docs Contradiction Toolkit$39 DIY · $199 Async Report
DIY Toolkit — $39 Async Report — $199

Guide · AI support bots & help-center docs

AI chatbot giving wrong answers? Check your help center first

Docs Contradiction Toolkit · Updated · Examples are illustrative, not customer stories.

A customer asks your support bot how long they have to request a refund. The bot says 60 days. Your policy is 30. Now you have a refund request you don't want to honor, a customer who has it in writing, and a bot you no longer trust.

The first move most teams make is to change the prompt or change the bot. Usually that's the wrong place to start. A docs bot doesn't know your business. It retrieves pieces of your help center and writes an answer from them. If those pieces disagree, are out of date, or leave out the condition that matters, the bot repeats the problem back to your customer in a confident voice.

This guide shows you how to trace one wrong answer back to the pages that caused it, fix the source, and check that the fix actually reached the bot.

The short answer

When a support bot built on your docs gives a wrong answer, the cause is usually one of four things in the source content:

  1. Two pages disagree. One says 30 days, another says 60.
  2. An old page is still live. Last year's policy sits next to this year's, and nothing tells the bot which one counts.
  3. The condition got separated from the claim. "Pro includes SSO" on one line, "on annual plans only" three paragraphs later.
  4. The answer was never written down. Your team answers it by hand every week. The docs don't mention it, so the bot guesses.

Model behavior matters too, but you can fix your pages today. Rule out the source first.

Why a bot gets it wrong when every page looks "fine"

A teammate reading your help center knows which page is the real policy. They know the 2023 FAQ is old. They know "Pro" means annual Pro. A retrieval bot knows none of that. It splits your pages into chunks, pulls the few chunks that look most similar to the question, and writes an answer from whatever it got.

Each page can be accurate and the set still wrong. The RAG guide from Unrag walks through this exact case: a 2022 refund document says 30 days, a 2024 one says 60, and without guidance the model "might pick one arbitrarily, average them incorrectly, or invent a reconciliation that doesn't actually exist in any source." It also notes that stale content "looks authoritative," because it was correct when it was written.

That last failure is the one that stings. For example: page A says refunds within 14 days, page B says 30 days, and the bot answers 21. The number 21 is on neither page. Nobody wrote it. The conflict did.

Intercom's help docs for its Fin AI agent include a "Fix contradicting content" feature that flags articles "at odds with one another" so the knowledge base can be "a single source of truth." A bot vendor shipping a contradiction finder is a good sign this is a common cause, not an edge case.

Diagnose one wrong answer, step by step

Start with one real wrong answer, not a full audit. One traced failure teaches you more than a spreadsheet of guesses.

Step 1: Capture the exact exchange

Copy the customer's question word for word, plus the bot's full reply. Wording matters: "can I get my money back" and "refund policy" can pull different chunks.

Step 2: Find what the bot actually read

If your bot shows citations or source links, open every one. If it doesn't, many platforms have a conversation or debug view that lists the sources used. If you can't see sources at all, search your help center for the key fact yourself (the number, the plan name, the menu label).

Step 3: Search for every version of the fact

Search for the fact itself, not the page title. For a refund window, search "refund", "money back", "30 days", "60 days", "cancel". Check:

  • Help center articles, including archived-but-public ones
  • Old blog posts and changelog entries
  • Pricing and terms pages
  • Any extra sources you uploaded to the bot: PDFs, website crawls, pasted snippets

Deleting a page from your site doesn't always delete it from the bot's source list.

Step 4: Classify it

Put the failure in one bucket:

What you found Bucket
Two live pages state different values Contradiction
An old page states a policy that changed Stale page
The right answer depends on a plan, region, or date that isn't next to the claim Buried condition
No page answers the question Missing answer
Pages are right and consistent, but the bot still quoted the old text Sync problem (the bot's copy is older than your site)

Step 5: Check whether the bot has your latest version

If your pages are correct and the bot is still wrong, check when the bot last re-crawled or re-synced its sources. Retrieval bots answer from an indexed copy of your content, not from the live page. Editing the page doesn't help until the bot re-reads it. For this case, see chatbot giving outdated answers.

Worked example: the 30-day vs 60-day refund

This example is made up to show the method. It isn't a real company.

Say your current refund policy page says: "Refunds are available within 30 days of purchase." An FAQ written before a policy change still says: "Not happy? You have 60 days to ask for a refund." A customer asks the bot, "How long do I have to get a refund?" The bot answers 60 days.

Running the steps:

  • Captured: the question and the "60 days" reply.
  • Sources: the bot cited the old FAQ. The FAQ is short, and its wording ("Not happy? ... ask for a refund") is close to how customers phrase the question, so it matched well.
  • Search: "60 days" turns up the FAQ and also a launch blog post that repeats the old policy.
  • Bucket: stale page, which is also a contradiction, because both versions are live.
  • Sync: not the issue. The bot read exactly what your site says.

The fix is not "tell the bot the refund window is 30 days" in the prompt. That leaves two public pages disagreeing for the next reader.

How to fix it

1. Pick one canonical page per fact

For each high-stakes fact (refund window, price, seat limits, cancel steps, SSO, data location), choose the one page that owns it. Write that choice down somewhere, even a simple list.

2. Rewrite the canonical page so each chunk stands on its own

Put the answer in the first sentence and keep the condition in the same sentence:

Refunds are available within 30 days of purchase for monthly and annual plans. After 30 days, refunds are not available.

Avoid "see above." A chunk that relies on another chunk loses meaning when retrieved alone.

3. Remove or point away the other versions

For each other page that states the fact, do one of these:

  • Delete it, or unpublish it if it has no other value.
  • Replace the stated value with a link to the canonical page: "See our refund policy."
  • If it has to stay for history (an old changelog entry), add a clear first line such as "This policy was replaced on [date]. Current policy: [link]."

Don't keep two live statements of the same number.

4. Remove it from the bot, not just the site

Check the bot's own source list. Remove old PDFs, pasted snippets, and crawled URLs that still carry the old value. Then trigger a re-sync or re-crawl.

5. Re-ask the question several ways

Ask the original question, then rephrase it: "can I get my money back," "refund after 6 weeks," "cancel and refund annual plan." If any version still returns the old value, go back to step 3. Something still has it.

Prevent it next time

  • Keep a short "money facts" list. Price, refund, cancel, seats, trial length, SSO, data location. These cause the most damage when wrong, so check them first.
  • Add "update docs + re-sync bot" to your release checklist. Put it next to whatever step already ships the change.
  • Give each canonical page an owner. Someone who gets pinged when the policy changes.
  • Keep a fixed set of probe questions. Re-run them after policy changes, so you catch regressions before customers do.

If you haven't launched a bot yet, run the help center AI readiness audit first.

What not to do

  • Don't patch the prompt and stop. A prompt rule can hide one symptom while the conflicting pages stay live.
  • Don't let the bot "split the difference." If two values exist, one is wrong. Decide which one.
  • Don't rewrite the whole help center first. Fix the facts that cost money or trust, then work outward.

FAQ

Why does my AI chatbot give wrong answers when my docs are correct?

Each page can be correct and the set can still conflict. A bot retrieves chunks from several pages. If an old page and a new page disagree, or a condition sits far from the claim, the bot can quote the wrong one or blend them.

Will a better prompt fix wrong answers?

It can help the bot handle conflicts more carefully, for example by telling it to cite sources and not invent reconciliations. It doesn't remove the conflicting pages. Fix the source, then tune the prompt.

I updated the article. Why does the bot still give the old answer?

Most retrieval bots answer from an indexed copy of your content. Until the bot re-crawls or re-syncs, it keeps reading the old version. Also check for copies of the old text in uploaded files or other pages.

Which pages should I check first?

Refund, pricing, cancellation, plan limits, trial terms, SSO, and data location. These are the answers where a mistake costs money or trust.

Do I need a new chatbot to fix this?

Usually not first. If the source pages disagree, a new bot reads the same pages. Clean up the source, then judge whether the bot itself is the problem.

If you want the method in one kit

If you'd rather not build the checklist and probe questions from scratch, the Docs Contradiction Toolkit packages this method as files you run yourself: a scorecard, an expanded checklist, anonymized worked examples, bot-probe prompts, and templates. It's a $39 one-time download. It isn't a chatbot or a subscription, and it doesn't promise to fix model behavior. It helps you find the contradictions, stale pages, and buried conditions in your public docs. Everything in this article works without it.

See what’s in the Docs Contradiction Toolkit

Sources