AI & MLLLMOpenAIPython

Making a Website Searchable with Embeddings and RAG

A website can contain years of useful material while still making it difficult to find an answer to a particular question. Keyword search helps locate a page, but a reader may need to combine information from several pages.

For a project that makes a WordPress blog searchable through questions, I adapted an OpenAI tutorial. The main practical change was to split the workflow into two stages: prepare the website content and embeddings, then use those saved representations to retrieve context for answers.

Preparing the website content

The first stage crawls the site, extracts text, creates embeddings, and writes the outputs. The retained README describes adjustments for WordPress, including ignoring some file URLs and comments or responses that did not belong in the intended content collection.

That filtering is more important than it first appears. A website contains navigation, repeated elements, media links, and other text that can distract from the material a question is actually about.

An embedding gives a passage a numerical representation that can be compared with a question's representation. The saved text and processed files let the answer stage reuse the prepared collection rather than recrawl it for every question.

Retrieving context before generating an answer

The second stage takes a question, retrieves relevant material using the embeddings, and supplies that context to the answer-generation step.

This is the useful idea behind retrieval-augmented generation: the application brings selected source material to the model at question time. The 2020 RAG paper provides a historical reference for that combination of retrieval and generation.

The experiment's repository was created in October 2023. That provides a dated anchor for this adaptation without assuming it was my first local attempt.

Why separate the stages

Crawling and answering have different jobs. The preparation stage needs to handle content boundaries and changes to the website. The answer stage needs to select useful passages and respond appropriately when the collection does not support an answer.

Separating them makes those problems easier to inspect. If an answer is poor, I can ask whether the relevant page was collected, whether its text was usable, whether retrieval selected it, and whether the model used it correctly.

Those are different failure points. Changing the final prompt will not fix a page missing from the collection.

What I would improve next

A stronger version would use a small set of questions with known supporting pages, retain the passages selected for each answer, and provide links back to the originals. It should also make an unsupported answer visibly different from one grounded in the site.

The practical lesson is that the quality of a website assistant depends on the content pipeline as much as the language model. The repository is an adapted tutorial, and its value is in making that pipeline concrete.

Code