Before you build an AI assistant, read this

The demo is always convincing. Someone asks a question, the assistant answers from the company handbook, and everyone in the room can see the value.

Then it goes to fifty staff who ask questions the demo never covered, against documents nobody curated, and confidence collapses within a fortnight. This pattern is common enough to be predictable, which means it is also designable-around.

Here is what we have learned building retrieval-augmented systems for clients in healthcare, financial services and logistics.

Retrieval quality is the whole game

Almost every failure blamed on "the AI hallucinating" is actually a retrieval failure. The model was given the wrong documents and answered from them faithfully. The generation step is rarely the problem; the search step almost always is.

Three things dominate retrieval quality:

Chunking. Splitting documents into fixed 500-token blocks is the default and it is usually wrong. A policy split mid-clause retrieves as a fragment that means something different from the whole. Chunk on document structure — sections, clauses, question-and-answer pairs — and keep the surrounding heading context attached to each chunk.

Hybrid search. Pure vector search misses exact identifiers: a policy number, a part code, an error string. Pure keyword search misses paraphrasing. Run both and combine the rankings. This single change usually improves retrieval more than upgrading the embedding model.

Reranking. Retrieve twenty candidates, then use a cross-encoder to rank them properly and pass the best five to the model. Cheap, fast, and consistently one of the highest-return additions.

Your documents are worse than you think

Every client believes their documentation is reasonable. It is worth checking before building anything.

What we typically find: three versions of the same policy with no indication which is current; documents referring to processes that changed two years ago; scanned PDFs with no text layer; critical knowledge that exists only in email threads and in one person's head.

An assistant built over that corpus will confidently cite the outdated policy, because it has no way to know it is outdated.

Budget for a content audit before the build. Sometimes the most valuable outcome of an AI project is discovering that your knowledge base needs work — and fixing it helps your staff whether or not the assistant ships.

Build the evaluation set before you build the assistant

This is the step most projects skip, and skipping it means you cannot tell whether a change made things better.

Collect 100 to 200 real questions from real users — support tickets, the queries people put to colleagues, the questions new joiners ask. Write the correct answer for each, with the source document. That is your evaluation set.

Now every change — a different chunking strategy, a new model, a revised prompt — can be measured. Without it, you are relying on someone's impression after trying five questions, which is how systems quietly get worse.

Run the evaluation in CI. Model providers update models; a prompt that worked can degrade without anyone touching your code.

Design for "I don't know"

The most valuable behaviour in a business assistant is declining to answer.

That requires deliberate engineering:

• A retrieval confidence threshold. If the best-matching documents score below a floor, say the answer is not available rather than generating from weak context.
• Explicit permission to refuse in the system prompt, plus examples of good refusals. Models default to helpfulness and will produce something plausible if not instructed otherwise.
• Citations on every claim, linked to the source. Users verify what matters, and a citation to an irrelevant document is a visible signal that something went wrong.
• A route to a human for anything the system declines, so a refusal is a handoff rather than a dead end.

An assistant answering 70 percent of questions accurately and declining the rest is far more useful than one that answers everything with 85 percent accuracy — because staff can trust the first and cannot trust the second.

Cost is an architecture decision

Inference cost per query is easy to ignore in a prototype and painful at full volume.

The levers, in rough order of impact:

• Model routing. Use a smaller, cheaper model for straightforward retrieval-and-summarise queries; escalate to a larger one only for complex reasoning. Most queries do not need the largest model.
• Prompt caching. If your system prompt and retrieved context repeat across queries, caching cuts cost substantially. On supported APIs this is a configuration change, not a rewrite.
• Retrieve less. Passing twenty chunks when five would do multiplies cost for no accuracy gain. Reranking lets you pass fewer with better results.
• Cache answers to repeated questions. In most organisations a small number of questions account for a large share of volume.

Model the cost per query during the prototype and design to it. Discovering the number after rollout is an unpleasant conversation.

Where a human must stay in the loop

Any output that carries financial, legal, medical or regulatory consequence needs a human decision point. Not because the model is unreliable in general, but because accountability has to sit somewhere a regulator or a customer can address.

Practically: the assistant drafts, a person approves. The assistant flags, a person decides. The assistant summarises a case, a person signs it off.

This is not a temporary arrangement pending better models. It is a governance design that stays.

A sensible sequence

1. Pick one narrow use case with a measurable current cost — time spent, tickets raised, errors made.
2. Audit the documents that would support it. Fix what is broken or pick a different use case.
3. Build the evaluation set from real questions.
4. Prototype in four to six weeks and measure against the evaluation set.
5. Pilot with ten users who will give honest feedback, and watch what they actually ask.
6. Then decide whether to widen the rollout or stop.

Steps 1 to 4 cost far less than a full build, and they tell you whether the full build is worth doing. We have advised clients to stop at step 4 more than once — and that was the right outcome, reached cheaply.

The uncomfortable conclusion

A lot of what gets scoped as an AI project is better solved by fixing a process, improving a search index, or writing the documentation that never got written.

That answer is less exciting and it saves a great deal of money. A partner willing to give it is worth more than one who will build whatever is asked for.