Skip to content

Preparing your knowledge base: web pages, PDFs, FAQs

An AI assistant is only as good as the content it reads. How to choose, clean up and organise web pages, sitemaps, PDFs and FAQs before you start indexing.

Knowledge and RAG6 min read

When an AI assistant gives a poor answer, the first reflex is often to blame the model. Most of the time the problem lies elsewhere: the information is not in the content, it appears twice in two different versions, or it is locked in a format that cannot be read. A RAG-based assistant repeats what it reads. The most valuable work therefore happens before indexing.

This article offers a method to prepare a client’s knowledge base, using the three kinds of sources blabla.bot accepts today: web pages, sitemaps and PDF files.

Start from the questions, not the documents

Before collecting files, list the questions visitors actually ask. Ask your client what their team hears on the phone and what keeps coming up in e-mails and reviews. Twenty to thirty questions are enough to begin.

For each question, find where the answer lives today:

  • on a page of the website;
  • in a document (manual, terms and conditions, brochure);
  • nowhere, except in someone’s head.

The third column is the most telling. A question with no written answer will get no answer from the assistant either. This is the moment to suggest your client writes the missing page.

Web pages: the simplest source

In blabla.bot you add the address of a page or of a sitemap.

A page address. The tool reads that page, then follows the links that stay within the same site, up to a maximum number of pages per source. This works well for a small, well-linked site.

A sitemap. The sitemap file lists the pages the site wants read. It is the most predictable method for a larger site: you know exactly which pages will be taken into account.

A few common-sense rules:

  • Respect what the site excludes. Pages disallowed by the robots.txt file or marked “noindex” are skipped. If a useful page is excluded by mistake, fix the site rather than trying to get around the rule.
  • Avoid pages behind a login. A page that asks for credentials returns an error and is not indexed.
  • Cut the noise. Legal notices, old news archives, tag pages: they dilute the results. A dedicated sitemap, limited to useful pages, often gives better answers.

After each crawl, the console shows the pages found, indexed and skipped, with the reason for each exclusion: blocked by robots.txt, outside the site, page in error, almost empty page, same content as another page, or maximum number of pages reached.

PDFs: useful, as long as they contain text

A lot of information lives in PDFs: catalogues, manuals, rules, price lists. blabla.bot reads them page by page, and answers cite the file name and page number.

Before uploading, check three things:

  1. The PDF contains text. Try selecting a sentence in the document. If you cannot, it is a scanned image: it will not be read, because character recognition (OCR) is not supported.
  2. The PDF is not password protected. A protected file is rejected with a clear message.
  3. The PDF is current. An old price list in a forgotten PDF will make the assistant say the wrong thing.

One practical point: by default, the file is deleted once indexed. Its passages stay available to the agent, but to re-index the document you will need to upload it again. Keep your PDF sources in a folder shared with your client.

Word documents and Excel files are not supported yet. In the meantime, export them to PDF or publish their content on a page of the site.

FAQs: write to be found

blabla.bot has no manual question-and-answer editor: an FAQ enters the knowledge base like everything else, as a web page or a PDF. It is often the most effective source, provided it is written well.

DoAvoid
One question per heading, phrased the way a visitor would ask itVague headings such as “Misc” or “Practical info”
A complete answer under each question that makes sense on its own“See above” or “as mentioned earlier”
Figures and conditions written out in fullAnswers that point to an image or a scanned table
A single version of each piece of informationThe same question with two different answers on different pages

The “answer that makes sense on its own” rule is essential. The assistant works on split passages: if the answer depends on a sentence three paragraphs earlier, it may not have that sentence in front of it.

Hunt down contradictions

This is the most common trap. The website promises delivery within 48 hours, the terms and conditions say three to five days, and last year’s PDF brochure says something else again. The assistant cannot tell which version is right.

Before indexing, run a quick pass over the sensitive topics: prices, lead times, opening hours, warranties, return conditions. For each one, keep a single source of truth and remove or fix the others.

Start indexing without surprises

Indexing has a cost, since every passage is turned into a vector by an embedding model. In blabla.bot that cost is shown before you confirm, as an estimate (“about X, at most Y”). For a sitemap, the estimate is based on the number of listed pages and an average page size; for a PDF, on the file’s text.

Then follow each source’s status on the “Knowledge” screen: queued, indexing, up to date, or error. If a new indexing run fails, the source becomes “Outdated” and the previous index stays in use: the assistant keeps answering from the earlier version.

A starter checklist

  1. List twenty to thirty real visitor questions.
  2. Find where each answer lives, and get the missing ones written.
  3. Prepare a sitemap limited to useful pages, or pick the right start page.
  4. Check that each PDF contains selectable text and is not protected.
  5. Remove contradictions on sensitive topics.
  6. Start indexing once you have read the cost estimate.
  7. Review the skipped pages and fix what needs fixing.
  8. Test the agent with your list of questions.

In short

A good knowledge base is not the biggest one, it is the clearest one. Start from visitors’ questions, keep one version of each piece of information, and make sure your documents really contain text. The assistant does the rest with what you give it.

To estimate what the agent will cost once live, the calculator on the pricing page gives you an order of magnitude based on your conversation volume.

Source

Tags

  • knowledge base
  • PDF
  • sitemap
  • FAQ