Chatbot That Only Answers From Your Documents — or Says I Don't Know
August 14, 2026 · 7 min read · by the hiy team
The short answer
To make a chatbot only answer from your documents and say "I don't know" otherwise, you need three layers, and they fail in a predictable order. A system-prompt instruction ("only answer from the context below") is the weakest — the model still fills gaps when the context is thin. A retrieval-score threshold is stronger, because if nothing clears the bar, nothing reaches the model and there is nothing to answer from. A visible receipt of what was searched is the layer a reader can check, because it separates a real "no match" from an answer nobody looked for. Then you test it: ask three questions you know are outside your material and count whether you get three refusals or three plausible paragraphs.
Layer one is the cheapest to add — one line of text — and the easiest to mistake for the whole job. A build that stops there still improvises.
How do you make a chatbot only answer from your documents?
You constrain what reaches the model, not just what you ask the model to do. The instruction layer is a request. The retrieval layer is a fact about what exists in the prompt. Only the second is enforceable, and only the third is checkable by the person reading the answer. Here is what each buys you.
| Layer | What it does | How it fails | How you catch that failure |
|---|---|---|---|
| 1. System-prompt instruction | Tells the model to answer only from the provided passages and to refuse otherwise | The model treats it as a preference. Give it three loosely related paragraphs and it will stitch a confident answer out of them | Ask something adjacent to your material but not in it. A leak here reads fluent and sourceless |
| 2. Retrieval-score threshold | Drops passages below a similarity cutoff, so a weak match returns an empty context | Set too loose, weak passages still arrive and layer 1 has to hold. Set too tight, real questions get refused | Ask something you know IS in your material, phrased differently than you wrote it. A false refusal means the cutoff is too tight |
| 3. Visible search receipt | Shows the reader which queries were run before the refusal | It does not prevent gaps. It only makes them auditable — and it is uncomfortable in public, which is the point | Read the receipt. If the searches look nothing like your question, retrieval is the problem, not the model |
Why a system-prompt instruction is not enough
Because a language model is a completion engine, and refusal is an unusual completion. You can write "if the answer is not in the context, say you don't know" in bold, three times, and it will hold on the easy cases — a question about a topic you have never touched, with an empty context window.
It breaks in the middle case, which is the common one. Someone asks about your refund policy. Retrieval returns your onboarding page, your terms summary and a post about client fit — all somewhat about money and commitments, none of them your refund policy. The model now has three plausible-looking passages and an instruction to use them. It writes a refund policy. It sounds like you. It is invented.
No amount of prompt engineering removes that, because the prompt cannot see that the passages are off-target. Something upstream has to.
How a retrieval threshold produces a real refusal
It makes refusal the only available output. If every candidate passage scores below your cutoff, the context is empty and the model has nothing to paraphrase. The refusal stops being an act of obedience and becomes an act of arithmetic.
Four practical notes if you are wiring this yourself:
- Set the cutoff empirically, not by taste. Run twenty questions you know are covered and twenty you know are not, then look at the score distributions. Your cutoff lives in the gap between them. If there is no gap, your chunking is the problem, not your threshold.
- A number that worked elsewhere will not transfer. Similarity scores are not calibrated across embedding models, chunk sizes or corpora. Change any of the three and you retune.
- Threshold on the best match, not the average. One strong passage among four weak ones is usually a real answer. Averaging hides it.
- Reword before you refuse. A question in the reader's words often misses material written in yours. Re-running the search with a couple of model-generated rewordings recovers the near-misses, so a refusal that survives every rewording is far more likely to be a genuine gap.
None of this requires a particular vendor. It is a property of how you assemble the prompt, so it works on any retrieval stack.
The third layer: show what was searched
A receipt turns a refusal from a claim into evidence. Without one, "I don't know" is indistinguishable from "I did not look." With one, the reader sees the question as it was sent, the rewordings that followed, and the fact that all of them came back empty — and can decide whether the gap is real or they should ask differently.
Write the refusal itself with the same care: a good one names what the system does cover, so the reader can redirect instead of leaving, and records the question rather than dropping it.
hiy ships this layer rather than leaving it to you to tune. When a hiy twin says it doesn't know, it shows the searches it ran first: the visitor's question, then its own rewordings. That receipt is on every plan, and no plan removes it. Grounded answers get the other half of the same contract — a citation that opens the whole passage and links back to the spot on your original page, so a reader can check the answer against the source instead of trusting the summary.
The case for why a refusal is worth designing at all, rather than how to build one, is in why AI twins should say "I don't know".
The three-question test, before you launch
Write down three questions you are certain are outside your material, then ask them. Three refusals means your layers are holding. Three plausible paragraphs means you have layer one only.
Choose the three deliberately, because easy misses prove nothing:
- Adjacent but absent. A topic in your field that you have genuinely never written about. This is the one prompt-only setups fail.
- Specific and unwritten. A number, price, date or policy you have never published. Models invent these most readily, because the shape of the answer is obvious even when the content is not.
- A confident false premise. "What did you say in your 2023 talk about pricing?" when there was no such talk. A well-built system refuses the premise. A weak one narrates the talk.
Then run the inverse test, which people skip: three questions you know your material answers well, phrased in words you did not use. If those get refused, your threshold is too tight and you have traded invention for uselessness.
In hiy, this runs in the test chat — a private sandbox before you publish, which counts against nothing — so you can find the leaks with nobody watching. The testing guide walks through it, and what it answers covers how scope boundaries are set.
The honest version
Refusal is not correctness. These three layers stop a chatbot answering from thin air; they do not stop it misreading a passage it genuinely retrieved. A confident, well-cited, subtly wrong paraphrase of your own document passes all three, because every check passed — something was retrieved, it scored well, the receipt looks clean. If your domain needs guaranteed-correct answers rather than sourced ones, no refusal layer gets you there.
Two more concessions worth making before you build:
- A threshold tuned loose enough to answer most questions will leak sometimes. There is no setting that is both maximally helpful and never wrong. Pick which error you would rather explain to a customer.
- Receipts do not fill gaps. They tell you a gap exists. Closing it — answering the question once so the system knows it from then on — is still your work. In hiy, unanswered questions queue up for exactly that, and your one-time answer teaches the twin permanently.
Where to start
If you are building this yourself, do layer two first. It changes behaviour rather than intent, and it will show you within an afternoon whether your chunking and embeddings can support any of it. Then add the receipt, because a refusal nobody can audit is a refusal nobody trusts.
If you would rather not build it, hiy does this part by default — cited answers, an honest "I don't know" in your tone, and the searches it ran shown underneath. You can build one free and run the three-question test before anyone else sees it.