An AI chatbot trained on your own data that answers only from it — or says “I don’t know”

The short answer

“An AI chatbot trained on your own data” usually means one thing in practice: the chatbot looks your documents up for each question and answers from what it found. To keep it honest you need three layers. A prompt rule is the weakest. A retrieval threshold is stronger. A visible receipt of what was searched is the one a reader can check. Then you test it: ask three questions you know are outside your material and count the refusals.

What “trained on your data” can mean

The phrase hides three different techniques.

1. Prompt stuffing. Your content, or a summary of it, is pasted into the model’s instructions. It works for a page of material and falls apart at ten. The tell: answers get vaguer as you add more.

2. Fine-tuning. The model’s weights are adjusted on your text. It is good at absorbing style and poor at facts. It blends your data into everything else it knows, cannot say where an answer came from, and cannot unlearn yesterday’s price.

3. Grounding (retrieval). Your content is indexed. For each question the most relevant passages are retrieved and the model must answer from them. It is the only one of the three where an answer can point to the passage behind it, and where an edit today changes tomorrow’s answer.

The rest of this post is about grounding, and how to make it hold.

The three layers that make a chatbot answer only from your documents

You constrain what reaches the model, not just what you ask of it. The instruction is a request. The retrieval layer is a fact about what is in the prompt. Only the second is enforceable, and only the third is checkable by the reader.

LayerWhat it doesHow it failsHow you catch that
1. Prompt instructionTells the model to answer only from the passages and refuse otherwiseThe model treats it as a preference. Given three loosely related paragraphs it stitches a confident answerAsk something adjacent to your material but not in it
2. Retrieval thresholdDrops passages below a similarity cutoff, so a weak match gives an empty contextToo loose: weak passages arrive. Too tight: real questions get refusedAsk something you know is covered, in words you did not use
3. Visible search receiptShows the reader which searches ran before a refusalIt does not close gaps, only makes them auditableRead the receipt. If the searches look nothing like the question, retrieval is the problem

Why a prompt instruction is not enough

Refusal is an unusual completion for a language model. The instruction holds on easy cases, such as a topic you never touched. It breaks in the common middle case. Someone asks about your refund policy. Retrieval returns your onboarding page, your terms summary and a post about client fit, none of them the refund policy. The model has three plausible passages and an instruction to use them, so it writes a refund policy. It sounds like you. It is invented. The prompt cannot see that the passages are off-target, so something upstream has to.

How a retrieval threshold produces a real refusal

If every candidate passage scores below the cutoff, the context is empty and the model has nothing to paraphrase. The refusal stops being obedience and becomes arithmetic.

  • Set the cutoff empirically. Run twenty questions you know are covered and twenty you know are not. Your cutoff lives in the gap between the two score distributions. No gap means your chunking is the problem.
  • A number from elsewhere will not transfer. Scores are not calibrated across embedding models, chunk sizes or corpora.
  • Threshold on the best match, not the average. One strong passage among four weak ones is usually a real answer.
  • Reword before you refuse. A question in the reader’s words often misses material written in yours. A refusal that survives several rewordings is far more likely to be a real gap.

The third layer: show what was searched

Without a receipt, “I don’t know” looks the same as “I did not look”. With one, the reader sees the question as sent, the rewordings, and that all of them came back empty. A good refusal also names what the system does cover, and records the question instead of dropping it.

hiy ships this layer. When a hiy twin says it does not know, it shows the searches it ran: the visitor’s question, then its own rewordings. That is on every plan. Why a refusal is worth designing at all is in why AI twins should say “I don’t know”.

Citations: a label or a receipt

Any chatbot will claim to know your content. A grounded one can prove it per answer. But “citation” covers four different things, and a tool sits on exactly one rung.

LevelWhat the citation opensWhat it proves
1. Names a fileNothing: “from your onboarding guide”A document with that name exists
2. Links a pageThe top of the source page or PDFThe document is real and yours, but you hunt for the paragraph
3. Opens the passageThe retrieved passage, in full, in placeThe model was handed this exact text
4. Opens the passage and deep-links the spotThe passage, plus a jump to that spot on the original pageWhat was used and where it sits in context

Level 1 is decoration. Level 2 is honest but lazy. From level 3 a citation is evidence, because you can hold the answer and its source side by side.

To test it, pick a question whose answer lives in one paragraph, ask it, and click the citation. Choose a needle, not a theme: a specific number or a policy exception. Time the click. Then ask something you have never written about, and watch what the tool attaches. A tool that must always cite will cite there too.

A citation proves where text came from, not that it was read correctly. Right passage with the wrong reading, a stale page still in the index, and two passages joined by an invented bridge all survive a perfect citation. It is chain of custody, not a verdict. The same goes for hiy. A badly retrieved passage is shown clearly under a confident sentence. Level 4 makes that catchable. It does not make it rare. AI support agent vs chatbot has more on telling the two apart.

The three-question test, before you launch

Write down three questions you are certain are outside your material, then ask them. Three refusals means the layers hold. Three plausible paragraphs means you have layer one only.

  1. Adjacent but absent. A topic in your field you have never written about. Prompt-only setups fail this one.
  2. Specific and unwritten. A number, price, date or policy you never published. Models invent these most readily.
  3. A confident false premise. “What did you say in your 2023 talk about pricing?” when there was no such talk.

Then run the inverse: three questions your material answers well, in words you did not use. If those get refused, the threshold is too tight.

In hiy this runs in the test chat, a private sandbox before you publish that counts against nothing. The testing guide walks through it, and what it answers covers scope.

Freshness, and putting it on your site

Your prices change. A grounded system re-indexes when you edit, and a correction you write overrides the older source, so the newest thing you said wins. A fine-tuned model needs retraining. hiy does not re-read your sources on a schedule: when you change a page, you re-add it.

On the page itself, the embed is one line and loads nothing until a visitor opens it:

<script src="https://hiy.ai/embed.js" data-twin="you" data-mode="bubble" async></script>

It can sit inline or as a floating bubble. The embed docs have the details. If the site is a company’s, you want a support-style agent with a neutral voice, and no public URL to embed. If the site is yours as a coach or creator, you want an AI twin that answers as you. Both ship from hiy.

The honest version

Refusal is not correctness. These layers stop a chatbot answering from thin air. They do not stop it misreading a passage it really retrieved. If your domain needs guaranteed-correct answers, no refusal layer gets you there.

  • A threshold loose enough to answer most questions will leak sometimes. Pick which error you would rather explain to a customer.
  • Receipts do not fill gaps. In hiy, unanswered questions queue up, and your one-time answer teaches the twin permanently.
  • hiy answers in text. There is no voice twin.

Where to start

If you are building this yourself, do layer two first. It changes behaviour rather than intent. Then add the receipt. The same layers have two more leaks to close for a product: numbers remembered instead of looked up, and stale pages outscoring new ones. How to stop a support agent inventing answers covers them.

If you would rather not build it, build one free and run the test before anyone else sees it, or try the demo twin.

Answers as you, or for your product. Cited, or an honest gap.

Start free
  • 14-day free trial
  • No card needed
  • Paste a URL to start