Skip to content

Measuring an AI agent's effectiveness: conversations, unanswered questions, improvements

Which indicators to follow to know whether an AI agent is helping, how to read them without fooling yourself, and a simple monthly routine to keep improving it.

The product5 min read

The agent has been live for a month. Your client asks the inevitable question: “So, is it working?” If you have prepared nothing, the answer will be a feeling. A feeling is not enough to justify the cost of an assistant, nor to know what to improve.

Measuring an AI agent’s effectiveness does not require a sophisticated dashboard. Mostly you need to know what you are looking for, which figures to watch and how to read them without drawing the wrong conclusions. This article offers a method, separating what blabla.bot shows today from what is planned.

Start with the goal

Before choosing indicators, agree with the client on what the agent should bring them. The most common goals are:

  • fewer repetitive requests (calls and e-mails about opening hours, lead times, conditions);
  • answers outside business hours;
  • guiding visitors to the right page or the right person;
  • collecting qualified contact requests.

Each goal calls for different indicators. An agent that mostly guides visitors does not need to “resolve” every conversation: it needs to get the visitor to the right place.

Indicators available today

In the blabla.bot console, the portfolio shows for each site:

IndicatorWhere to find itWhat it tells you
Conversations over 30 daysPortfolioActual usage by visitors
Billed cost over 30 daysPortfolioWhat the agent consumed in credits
Monthly budget and spendPortfolio, CreditsWhere the site stands against its cap
Agent statusPortfolioOnline, to install, importing, low credits, paused

Test conversations run from the simulator are not counted in the number of conversations: that figure reflects visitors’ usage, not your own trials. Their cost, however, is still spending: when tests consume credits, it appears in the billed cost and in the site’s consumption.

One simple calculation follows: the average cost per conversation, dividing the 30-day billed cost by the number of conversations. It is the figure to watch to check that the agent stays profitable for you and affordable for the client. Bear in mind that a month heavy in testing inflates it slightly.

Indicators on the way

Several screens are planned in the console’s “Conversations” section:

  • the conversation history, to read back exchanges;
  • unanswered questions, the ones where the agent found no passage relevant enough;
  • escalations to a human, when the visitor left a contact request.

Unanswered questions are already recorded by the platform, with the reason and the score reached. They will feed that screen as soon as it ships.

Reading the figures without fooling yourself

A few traps come up often.

Low volume is not failure. If the site gets few visits, the agent will have few conversations. Compare volume with site traffic, not with an abstract expectation.

High volume is not success. Many conversations can mean the site is poorly organised and visitors cannot find information on their own. That is useful to know, but it is not necessarily to the agent’s credit.

An unanswered question is not a mistake. Often it is the agent doing its job by refusing to invent. The real signal is repetition: the same unanswered question every week points to missing content.

Beware of averages. A stable average cost per conversation can hide a few very long conversations. If the cost suddenly rises, check the per-site consumption on the “Credits” screen.

A five-step monthly routine

What improves an agent is a regular routine rather than a big yearly audit. Here is a method you can apply right away.

  1. Note the portfolio figures: conversations and billed cost over 30 days, spend against budget.
  2. Replay your reference question set in the simulator. Keep a fixed list of about twenty real questions, including a few with no answer on the site. Note the ones whose answer got worse or better.
  3. Gather the client’s feedback: is their team getting fewer repetitive questions? Which questions still arrive by phone or e-mail?
  4. Fix the content: write the missing page or FAQ answer, remove contradictions, then re-index the sources concerned.
  5. Adjust the behaviour if needed: sharpen the mission, add or remove a forbidden topic, revisit the welcome message.

Step two is the most underrated. A reference question set replayed every month gives you a stable measure of quality, independent of traffic. It also lets you check that a re-index or a change of AI option has not broken anything.

The simulator as a measuring tool

The simulator is not just for launch. For each answer, the “Why this answer” panel shows the passages kept, the best relevance score against the threshold, the input and output tokens used, and the cost of the answer. It is a valuable tool to understand a disappointing answer or to estimate how a change of AI option affects cost.

Reporting to the client

A short monthly report is often enough:

  • conversation volume and its trend;
  • the month’s cost compared with the planned budget;
  • the three pieces of content added or fixed, and why;
  • the questions still to address, with what the client needs to provide.

This format shows the agent is being looked after, and it involves the client in improving it: they are often the one holding the missing information.

In short

Measuring an agent’s effectiveness means linking a few simple figures to a clear goal. Today, conversation volume, billed cost and per-site budget give a solid base, complemented by a reference question set replayed in the simulator. The history and unanswered questions screens will add to that picture.

To estimate an agent’s cost for an expected conversation volume, use the calculator on the pricing page.

Tags

  • metrics
  • conversations
  • continuous improvement
  • agency