Retrieval before fine-tuning: making an LLM answer from your own documents

A common first request when a team starts with large language models is: "Can we train it on our documents?" The instinct is understandable. The model does not know your policies, your product or your contracts, so teaching it seems like the obvious move. In most cases, though, the better first step is not training at all. It is retrieval: finding the right passages at the moment a question is asked and handing them to the model to read.

This essay walks through why, and then builds a small but real system end to end: an assistant that answers staff questions about a company's HR policies. The example is deliberately ordinary. Most useful AI systems are.

Two ways to give a model knowledge

Fine-tuning adjusts the model's weights using examples. It is good at teaching a style, a format, a tone or a narrow behaviour: always answer in this JSON shape, always write like our support team, always classify tickets into these twelve categories. It is a poor way to store facts that change. Every policy update means another training run, you cannot easily tell which fact the model learned from which document, and the model can still blend what you taught it with what it already believed.

Retrieval-augmented generation, usually shortened to RAG, leaves the model's weights alone. When a question arrives you search your own documents for the passages most likely to contain the answer, place them in the prompt, and instruct the model to answer from them. Update a document and the next answer reflects it. Delete a document and it stops being used. You can show the reader exactly which passage an answer came from.

Use retrieval for knowledge. Use fine-tuning for behaviour.

Most "train it on our documents" requests are knowledge problems. That is why retrieval comes first.

The worked example: an HR policy assistant

Imagine a company of a few hundred people with forty policy documents: leave, travel, expenses, remote work, equipment, conduct. Staff ask the same questions every week. How many days of compassionate leave do I get? Can I claim a taxi after a late shift? The HR team answers them by hand. The goal is an assistant that answers correctly, points to the policy it used, and says plainly when the policies do not cover a question.

Before writing any code, write down three things, the same way you would start from the decision, not the model. Who asks: any employee, on a phone or a laptop. What a good answer looks like: correct, short, with the policy name and section. What a bad answer costs: a wrong leave entitlement is a real problem for a real person, so the system must prefer "I don't know, ask HR" over a confident guess.

Step 1: Collect and clean the documents

Gather the current version of every policy, and only the current version. Old drafts in the index are the most common cause of confidently wrong answers. Convert everything to plain text with its structure kept: headings, numbered clauses and tables. Record, for each document, its title, its owner and its last-updated date. You will want all three in the answer later.

Step 2: Split the documents into passages

Models answer from passages, not whole documents, so each document is split into chunks. This step quietly decides how good the whole system will be.

• Split on structure, not on character counts. A policy clause, with its heading, is a natural unit. Cutting a table or a numbered list in half produces fragments that mean nothing on their own.
• Keep chunks small enough to be specific. A few paragraphs is a good starting size. A chunk that covers five topics will be retrieved for all five and be precise about none.
• Carry context into every chunk. Prefix each chunk with its document title and section heading, so "Up to 5 days" becomes "Leave Policy, Section 4: Compassionate leave. Up to 5 days...". Without that prefix, a chunk can be perfectly relevant and still useless.

Step 3: Index the passages two ways

Each chunk is turned into an embedding: a list of numbers that captures its meaning, so that passages about the same idea sit close together even when they use different words. "Bereavement" and "compassionate leave" end up near each other. This is what makes semantic search work.

Embeddings have a known weakness: exact terms. Policy codes, form numbers, names and amounts are often matched poorly by meaning alone. So index the same chunks a second way, with ordinary keyword search, and run both searches for every question. Combining the two result lists, often called hybrid search, catches what either would miss on its own.

Step 4: Retrieve, then rerank

For each question, take the top results from both searches, perhaps twenty in total, and then put them through a second, more careful step that scores how well each passage actually answers this question. That step is called reranking. It is slower per passage, which is why it runs only on the shortlist. Keep the best three to five passages for the prompt.

Step 5: Ask the model to answer from the passages, and only from them

The prompt does three jobs. It gives the model the question. It gives it the passages, each labelled with its source. And it sets the rules:

• Answer only from the passages provided.
• Name the policy and section each part of the answer comes from.
• If the passages do not contain the answer, say so and suggest contacting HR. Do not guess.

That last rule is the most important sentence in the whole system. A retrieval assistant that admits "the policies don't cover this" is far more trustworthy than one that improvises, and in an HR setting trust is the entire product.

Step 6: Show your sources

Display the policy names and sections under every answer, linked to the documents. Readers check the important answers themselves, which is exactly what you want, and when an answer is wrong the citation shows you instantly whether retrieval found the wrong passage or the model misread the right one. Those are different bugs with different fixes.

Measure before you tune

Most teams start adjusting chunk sizes and prompts by feel at this point. Don't. Build a small test set first. (We go deeper on this in Test your AI feature like you would test a payments flow.)

Ask HR for fifty real questions from the last few months, including the awkward ones: the vague question, the question with a typo, the question the policies genuinely do not answer. For each, write down the correct answer and the policy section it comes from. Then run every question through the system and record two separate scores:

• Retrieval: was the correct section among the passages handed to the model?
• Answer: was the final answer correct, cited, and honest about gaps?

Separating the two is what makes the numbers useful. If retrieval found the right passage but the answer was wrong, work on the prompt. If retrieval missed the passage, no prompt will save you; work on splitting, the context prefixes, or the search itself.

Where these systems usually go wrong

• Stale documents. The old travel policy is still in the index and wins half the searches. Fix the data pipeline before touching anything else.
• Chunks without context. The right clause is retrieved, but without its heading the model cannot tell which policy it belongs to.
• Exact terms missed. A question about "form HR-12" retrieves nothing useful because only semantic search was used.
• Answers without citations. Errors become invisible, and readers have no way to check.
• No "I don't know". The model fills gaps with plausible-sounding policy that does not exist.
• Access control forgotten. If some documents are confidential, the index must respect the same permissions as the documents. A retrieval system can leak anything it can retrieve.

When fine-tuning does earn its place

Fine-tuning is not the enemy; it is the second tool. Reach for it when the model consistently finds the right facts but delivers them in the wrong shape: too long, wrong tone, inconsistent format, poor at a narrow classification your business depends on. It also helps when you need a small, cheap model to perform one job that a large model does well, so you can run it faster and at lower cost. In mature systems the two are often combined: retrieval supplies the facts, and a fine-tuned model writes the answer in the house style.

What to do this week

1. Pick one narrow body of documents where people already ask repeat questions.
2. Collect fifty real questions and their correct answers before writing code.
3. Build the simplest pipeline above: structured chunks with context, hybrid search, a strict prompt, visible citations.
4. Score retrieval and answers separately, fix the larger problem first, and repeat.

You will learn more from that first honest measurement than from any amount of model shopping. And you will have built something your colleagues can trust, which is the only kind of AI system worth putting in front of them.