How to Test an AI Support Agent Before Customers See It: A 20-Question Protocol

The short answer#

Test an AI support agent with twenty questions in five kinds, four of each: questions your docs answer, in the docs' own words; the same questions reworded the way a customer would type them; questions adjacent to your product that the docs do not cover; questions built on a false premise; and questions that are hostile, off-topic or trying to make it sell. Grade every answer one of four ways — cited and correct, declined correctly, declined wrongly, or answered wrongly — and fix the source that produced the grade, never the answer itself. Run it in a sandbox that gives production answers and records nothing, and run it again after every fix. The pass mark is not twenty out of twenty; it is zero answered-wrongly, because that is the only grade a customer will screenshot.

Twenty is a floor. Passing it proves the controls hold on twenty questions. Real tickets from last month are the best source of the twenty.

Why test before launch rather than watch the first week?#

Because the first invented answer is public, in your product's name, and screenshot-ready. A wrong answer from a human support agent is a bad day. A wrong answer from the product about the product reads as the product lying. The customer has no reason to think a machine got it wrong and every reason to think the company put it in writing.

There is also a quieter reason. Once customers are asking, you will be tempted to fix answers — a prompt tweak here, a phrase there — because it is fast. Before launch, with nobody waiting, you fix sources, which is slower and the only thing that lasts.

The five kinds of question#

Four of each. The first two kinds prove the agent is useful; the last three prove it is safe. Teams usually test only the first kind, ship, and meet the third kind on day one.

How to grade an answer#

Four grades, and each one points at a different thing to fix.

Cited and correct. The answer matches the docs and the citation opens to the passage that supports it. Open it every time; a citation that names the right page but not the right passage is a label, not a receipt.

Declined correctly. The docs do not cover it and the agent said so in plain words, then offered a person. This is a pass. It is also a gap to close later — the question is real, and answering it once turns it into a source.

Declined wrongly. The docs do cover it and the agent refused anyway. Almost always a retrieval miss on a rewording, or a threshold tuned too tight after an earlier leak. The fix is on the source side: a heading that matches how customers ask, or the answer moved into the first two sentences of its passage. How to write help docs an agent can answer from is the checklist.

Answered wrongly. The agent produced something the docs do not support — an invented integration, a number from memory, an explanation of a premise that was false. This is the only failing grade, and one is too many. Trace it: was retrieval thin (scope too loose), was a number reconstructed from prose (it should be a list row), did an old page outscore the new one (a contradiction, or a correction needed), or did the tone round up (a persona that should not exist on a support agent)? How to stop a support agent inventing answers maps each cause to its control.

What the sandbox has to do#

A test is only a test if the answers you get are the answers a customer would get and nothing you do leaves a trace. Four properties:

• Same answers as production. Same retrieval, same citations, same decline wording. A "test mode" that is more careful than the live agent tests nothing.
• Nothing recorded. No message allowance used, no line in your analytics, no entry in the queue of unanswered questions. You need to ask the same awkward question twenty times.
• Actions rehearsed, not performed. If the agent can notify someone or call your endpoint, the sandbox should show you it would have, without doing it.
• The customer's view, at a customer's width. Including the handoff panel and the email box, on a phone-sized screen, because that is where half your customers will meet it.

In hiy this is the Sandbox: a private chat with the same answers, citations and honest-gap treatment a visitor gets, where test messages count against nothing and any action is rehearsed rather than run. The full-page preview beside it shows the widget at phone, tablet and desktop width, on a plain backdrop, with the handover form you can fill in yourself. It works on a draft agent, before anything is published, and a Support Agent's See it answer button mounts the real widget with a ten-minute token before you have written any server code.

Building the set from real tickets#

The best twenty questions are the ones customers already asked. Take last month's tickets and sort them into the five kinds:

• The ones the docs answered, kept in the customer's exact words — that is your reworded set, and it is better than anything you would invent.
• The ones the docs did not answer — your adjacent but absent set. These are the questions that will reach the agent first.
• Any ticket that asserted something untrue — a false premise, in the wild.
• Anything rude, off-topic, or asking which plan to buy — your hostile set.

If you have no tickets yet, write the twenty yourself, then replace them with real ones after the first fortnight. The queue of questions the agent could not answer is a test set that writes itself.

After launch: the test that never stops#

The protocol does not end at launch; it changes shape. Every question the agent declines and records is an adjacent but absent question with a real customer behind it, and answering it once is both the fix and the next test. hiy's queue only fills with genuine declines — matched against the wordings the model actually produces rather than inferred from weak retrieval — so what you get is a list worth working through.

And resist reading the rates too early. A deflection or escalation percentage off a handful of conversations swings five points when one customer changes their mind, which is why hiy shows neither until twenty conversations have happened in the window. Until then, the queue is the number.

The honest version#

Twenty questions prove the controls hold on twenty questions. The twenty-first can still leak, and a well-cited, subtly wrong paraphrase of a real passage will pass every grade above because everything about it looks right — something was retrieved, it scored well, the citation opens. The only defence against that one is the citation being readable, so the customer can catch what you did not.

And a set you wrote yourself tests your imagination, not your customers. Replace it with real tickets as soon as you have them.

Where to start#

Write the twenty today, from tickets if you have them. Ask them in a sandbox. Count the answered-wrongly grade first — if it is not zero, nothing else about the launch matters yet. If you are building on hiy, the Sandbox is one click from any agent page and the Support Agent starts on the strict scope the third and fourth kinds of question are designed to break.