Skip to content

Keeping the knowledge base up to date: manual and automatic re-indexing

When a site changes, the assistant must follow. When to re-index, what it costs, what is not billed again, and how the automatic re-indexing option works.

Knowledge and RAG5 min read

An AI assistant answers from the knowledge it has been given. On launch day, that knowledge is current. Three months later, the client has changed their opening hours, added a service, withdrawn an offer and published a new FAQ page. If nobody has updated the knowledge, the assistant hands out outdated information with the same confidence as on day one.

It is one of the most common risks, and one of the easiest to avoid. This article explains how blabla.bot keeps knowledge current, what a re-index costs, and how to organise that work for your clients.

Why knowledge goes stale

During indexing, blabla.bot reads the client’s pages and PDFs, splits them into passages and turns them into vectors, a representation that makes it possible to find passages close to a question. The assistant then searches this copy, not the live site.

That copy does not change on its own. A page edited on the site stays in its old version on the assistant’s side until the next indexing run. Hence the importance of re-indexing at the right time.

Source types and how to update them

SourceHow it is readHow to update it
Web pageThe agent explores the site starting from that pageRe-index the source
SitemapThe agent reads the pages listed in the fileRe-index the source
PDF fileText is extracted page by pageUpload the file again

A sitemap is a file in a standard format that lists the addresses of a site’s pages. It is often the simplest way to hand a whole site to the agent.

An important point about PDFs: by default, the file is deleted once indexed. Only the extracted text is kept in the knowledge. To update a PDF, you therefore upload its new version.

Manual re-indexing

On a site’s “Knowledge” screen, you can re-index a single source, or every source of the agent with “Re-index all”.

An estimate before you start

No indexing starts without your confirmation. Before launch, the console shows a cost estimate in the form “about X, at most Y”. blabla.bot calculates it with the same formula as the bill. When a reliable estimate is impossible, for example for an unreadable address, the console says so instead of showing a made-up figure.

The estimate is compared with your available balance. If the balance looks tight or insufficient, a warning appears with a link to the “Credits” screen. You remain free to go ahead.

The real cost afterwards

Once indexing finishes, the console shows its actual cost next to the estimate, with the date and the number of pages reused.

Unchanged pages are not billed again

This is what makes regular re-indexing affordable. For each page, blabla.bot keeps a fingerprint of its content. During a re-index, a page whose fingerprint has not changed reuses its existing vectors, with no new call to the provider and therefore no new cost. Only changed or new pages are billed.

There is one exception: if the embeddings model or the splitting method changes, every page is processed again.

What it costs

In the current configuration, indexing is billed at €0.20 per million tokens (the embeddings service price, multiplied by two). A 50-page site with around 4,000 characters per page comes to roughly 50,000 tokens, so its first indexing costs about one cent. A re-index where only a few pages have changed costs even less.

If indexing fails

If a re-index does not complete, for instance for lack of credits, the previous version of the knowledge stays in service. The assistant keeps answering with the information it had, and the console flags the source concerned.

Automatic re-indexing

For web sources, blabla.bot can also re-index automatically, at a frequency you choose source by source: weekly or monthly.

It follows a few cautious rules:

  • It is off by default on every source. A recurring expense should only start with an explicit choice.
  • You see the cost before turning it on: an estimate per run and per month.
  • The first run happens one interval after you turn it on, never immediately.
  • A run is skipped if your credits are insufficient. The source keeps being served in its current version, and the console shows that the run was skipped. The visitor notices nothing.
  • It does not apply to PDFs, since the file is not kept.

At the time of writing, this feature is still switched off on the production platform; it will be opened gradually. Until then, manual re-indexing remains the rule.

Organising updates for your clients

Technology alone is not enough: you need to know when content changes. Here is a simple setup.

  1. Ask the client to tell you about every significant change: prices, hours, a new offer, an exceptional closure. Write it into your maintenance contract.
  2. Re-index right after every site update you make yourself. That is when you know exactly what changed.
  3. Schedule a monthly re-index for sites that change often, such as stores or news sites. Since unchanged pages are not billed again, the cost stays low.
  4. Remove obsolete sources: an old PDF brochure can contradict the site.
  5. Check in the simulator after a major update: ask the question the change affects and check the answer and its source.

In short

Up-to-date knowledge makes for a trustworthy assistant. Re-index after every change, take advantage of unchanged pages not being billed again, and upload revised PDFs. Automatic re-indexing will complement this work for sites that change often.

More questions about knowledge and indexing are answered in the FAQ.

Sources

Tags

  • knowledge
  • re-indexing
  • RAG
  • costs