Measuring an AI agent's effectiveness: conversations, unanswered questions, improvements
Which indicators to follow to know whether an AI agent is helping, how to read them without fooling yourself, and a simple monthly routine to keep improving it.
The product5 min read
The agent has been live for a month. Your client asks the inevitable question: “So, is it working?” If you have prepared nothing, the answer will be a feeling. A feeling is not enough to justify the cost of an assistant, nor to know what to improve.
Measuring an AI agent’s effectiveness does not require a sophisticated dashboard. Mostly you need to know what you are looking for, which figures to watch and how to read them without drawing the wrong conclusions. This article offers a method, separating what blabla.bot shows today from what is planned.
Start with the goal
Before choosing indicators, agree with the client on what the agent should bring them. The most common goals are:
- fewer repetitive requests (calls and e-mails about opening hours, lead times, conditions);
- answers outside business hours;
- guiding visitors to the right page or the right person;
- collecting qualified contact requests.
Each goal calls for different indicators. An agent that mostly guides visitors does not need to “resolve” every conversation: it needs to get the visitor to the right place.
Indicators available today
In the blabla.bot console, the portfolio shows for each site:
| Indicator | Where to find it | What it tells you |
|---|---|---|
| Conversations over 30 days | Portfolio | Actual usage by visitors |
| Billed cost over 30 days | Portfolio | What the agent consumed in credits |
| Monthly budget and spend | Portfolio, Credits | Where the site stands against its cap |
| Agent status | Portfolio | Online, to install, importing, low credits, paused |
Test conversations run from the simulator are not counted in the number of conversations: that figure reflects visitors’ usage, not your own trials. Their cost, however, is still spending: when tests consume credits, it appears in the billed cost and in the site’s consumption.
One simple calculation follows: the average cost per conversation, dividing the 30-day billed cost by the number of conversations. It is the figure to watch to check that the agent stays profitable for you and affordable for the client. Bear in mind that a month heavy in testing inflates it slightly.
Indicators on the way
Several screens are planned in the console’s “Conversations” section:
- the conversation history, to read back exchanges;
- unanswered questions, the ones where the agent found no passage relevant enough;
- escalations to a human, when the visitor left a contact request.
Unanswered questions are already recorded by the platform, with the reason and the score reached. They will feed that screen as soon as it ships.
Reading the figures without fooling yourself
A few traps come up often.
Low volume is not failure. If the site gets few visits, the agent will have few conversations. Compare volume with site traffic, not with an abstract expectation.
High volume is not success. Many conversations can mean the site is poorly organised and visitors cannot find information on their own. That is useful to know, but it is not necessarily to the agent’s credit.
An unanswered question is not a mistake. Often it is the agent doing its job by refusing to invent. The real signal is repetition: the same unanswered question every week points to missing content.
Beware of averages. A stable average cost per conversation can hide a few very long conversations. If the cost suddenly rises, check the per-site consumption on the “Credits” screen.
A five-step monthly routine
What improves an agent is a regular routine rather than a big yearly audit. Here is a method you can apply right away.
- Note the portfolio figures: conversations and billed cost over 30 days, spend against budget.
- Replay your reference question set in the simulator. Keep a fixed list of about twenty real questions, including a few with no answer on the site. Note the ones whose answer got worse or better.
- Gather the client’s feedback: is their team getting fewer repetitive questions? Which questions still arrive by phone or e-mail?
- Fix the content: write the missing page or FAQ answer, remove contradictions, then re-index the sources concerned.
- Adjust the behaviour if needed: sharpen the mission, add or remove a forbidden topic, revisit the welcome message.
Step two is the most underrated. A reference question set replayed every month gives you a stable measure of quality, independent of traffic. It also lets you check that a re-index or a change of AI option has not broken anything.
The simulator as a measuring tool
The simulator is not just for launch. For each answer, the “Why this answer” panel shows the passages kept, the best relevance score against the threshold, the input and output tokens used, and the cost of the answer. It is a valuable tool to understand a disappointing answer or to estimate how a change of AI option affects cost.
Reporting to the client
A short monthly report is often enough:
- conversation volume and its trend;
- the month’s cost compared with the planned budget;
- the three pieces of content added or fixed, and why;
- the questions still to address, with what the client needs to provide.
This format shows the agent is being looked after, and it involves the client in improving it: they are often the one holding the missing information.
In short
Measuring an agent’s effectiveness means linking a few simple figures to a clear goal. Today, conversation volume, billed cost and per-site budget give a solid base, complemented by a reference question set replayed in the simulator. The history and unanswered questions screens will add to that picture.
To estimate an agent’s cost for an expected conversation volume, use the calculator on the pricing page.
Tags
- metrics
- conversations
- continuous improvement
- agency
On the site
Read next
Knowledge and RAG
Why a good assistant knows how to say "I don't know"
Relevance threshold, human handoff, unanswered questions: why a reliable AI assistant would rather admit it does not know than invent a plausible answer.
· 5 min read
The product
Testing an AI agent before launch: the simulator and a method
How to test an AI agent before installing it: a simulator that reproduces the widget, an explanation panel, and a question set that covers the difficult cases.
· 5 min read
Costs and AI models
Monthly budget per site and alerts: avoiding unpleasant surprises
Monthly cap per site, alert threshold, balance alert: how to set the blabla.bot safeguards so that a sudden traffic spike does not eat into your margin.
· 5 min read