Why a good assistant knows how to say "I don't know"
Relevance threshold, human handoff, unanswered questions: why a reliable AI assistant would rather admit it does not know than invent a plausible answer.
Knowledge and RAG5 min read
A visitor asks a hotel’s assistant whether dogs are allowed. The website says nothing about it. Two answers are possible. The first: “Yes, pets are welcome.” It is friendly, reassuring, and possibly wrong. The second: “I don’t have that information. You can contact the team to check.” It is less appealing, but it misleads nobody.
Left to itself, a language model tends to produce the first one. It is built to generate plausible text, not to check that it actually has the information. For a business, that plausible but invented answer is the worst case: a disappointed guest on arrival, a promise the team never made, trust damaged.
This article explains how an assistant grounded in a business’s content can recognise that it does not know, and what blabla.bot puts in place to make that happen.
The relevance threshold at the heart of it
A quick reminder: when a visitor asks a question, the assistant searches the knowledge base for the closest passages. Each passage gets a relevance score between 0 and 1. The higher the score, the closer the passage is to the question.
The relevance threshold is the minimum score a passage needs to be used. In blabla.bot:
- above the threshold, passages go to the model, which writes its answer from them and cites them;
- below the threshold, no passage goes to the model. Instead it gets an explicit instruction: say it does not have the information, offer contact with the team if there is one, and invent nothing.
The instruction also covers small talk: the assistant still replies to “Hello” or “Thank you”, even when no passage matches.
A threshold for each model
A relevance score only means something for a given embedding model. A score of 0.8 can be excellent with one model and mediocre with another. That is why blabla.bot calibrates the threshold for the embedding model in use, on a set of French questions and passages, rather than applying a universal value.
Three situations to tell apart
Not every missing answer is the same. blabla.bot distinguishes several cases.
| Situation | What the assistant does | Human handoff offered |
|---|---|---|
| Some passages are above the threshold | Answers and cites its sources | No |
| The agent has knowledge, but nothing close enough | Says it does not know | Yes, if a contact is set |
| The agent has no indexed knowledge yet | Answers from its settings only, with no passage | No |
The third row needs a word of explanation. An agent that has not read anything yet is not failing: it is simply not set up yet. Making it offer a human contact on every message would annoy visitors and help nobody. It is the agency’s job to give it knowledge before going live.
Human handoff: turning a dead end into a contact
When the assistant does not know and a contact e-mail has been set, the widget offers the visitor a form to leave their e-mail address and a message. The visitor leaves with a way forward, not a dead end.
When a contact is set, visitors can also ask to talk to a human themselves through a link in the widget, even after the assistant has answered, if they are unsure about the answer.
Unanswered questions: a goldmine
Every question that falls below the threshold, or is asked to an agent with no knowledge, is recorded with the reason, the best score reached and the threshold in force. Test conversations run from the console are excluded so they do not skew the data.
These questions are valuable: they show exactly what visitors are looking for that the content does not cover. A dedicated unanswered questions screen is planned in the console, under “Conversations”. In the meantime, the simulator already shows, question by question, why the agent answered or not.
Watching the threshold at work in the simulator
On the “Test” screen, every answer can be explained. The “Why this answer” panel shows:
- the search status (passages found, below the threshold, no knowledge);
- the best relevance score reached, compared with the threshold;
- the selected passages, numbered, with their score and whether they were cited.
In practice, if the agent says “I don’t know” to a question you thought was covered, look at the best score. If it is close to the threshold, the page probably exists but words the information differently: rephrase it, or add an explicit question to the site’s FAQ. If it is very low, the information simply is not in the knowledge base.
The limits, honestly
This setup greatly reduces the risk of invented answers. It does not remove it entirely.
It is an instruction, not a guarantee. When nothing passes the threshold, the model is told to say it does not know. No model provider guarantees that an instruction will be followed every time. That is why the widget reminds visitors that the assistant can make mistakes.
Relevant but incomplete passages. A passage can pass the threshold without holding the whole answer. The assistant then has partial context. The platform rules tell it not to fill the gaps itself, but this is a case only careful testing will reveal.
Some noise. A simple “Hello” can fall below the threshold on an agent that has knowledge. The assistant replies politely, but the message may be recorded as unanswered. Keep that in mind when you read the data.
What you can do
- Set a contact e-mail that someone reads, so every “I don’t know” leads somewhere.
- Test with real questions, including questions whose answer is not on the site, to check the agent knows when to stop.
- Use the simulator’s explanation to understand edge cases.
- Enrich the content based on the questions that remain unanswered.
In short
An assistant that says “I don’t know” at the right moment is not a faulty assistant. It is a reliable one. The relevance threshold, the explicit instruction and the human handoff turn an impossible answer into a useful contact, and unanswered questions tell you exactly which content to add.
Common questions about answer reliability are gathered in the FAQ.
Tags
- RAG
- relevance threshold
- reliability
- human handoff
On the site
Read next
Knowledge and RAG
RAG explained simply: an assistant that answers from your own content
RAG lets an AI assistant answer from a business's own content instead of its memory. How it works step by step, where it falls short, and good habits.
· 5 min read
The product
Measuring an AI agent's effectiveness: conversations, unanswered questions, improvements
Which indicators to follow to know whether an AI agent is helping, how to read them without fooling yourself, and a simple monthly routine to keep improving it.
· 5 min read
Knowledge and RAG
Preparing your knowledge base: web pages, PDFs, FAQs
An AI assistant is only as good as the content it reads. How to choose, clean up and organise web pages, sitemaps, PDFs and FAQs before you start indexing.
· 6 min read