AI in production: RAG, evals and prompt injection

There is a predictable pattern in AI projects: the demo works in two weeks, and the product does not ship in six months.

It is not a lack of technical ability. It is that the demo and the product require different things, and the three things the product requires are precisely the least glamorous.

First: retrieval that works#

When a RAG system answers wrongly, the team's instinct is to change models. In the overwhelming majority of cases, the problem is not the model: it is retrieval.

If the right passage did not make it into the context, no model can save you.

The causes, in order of frequency:

Bad chunking. You split by a fixed character count and cut the table in half, or separated the question from the answer. Split by structure (section, paragraph) with some overlap. And store the section title alongside the chunk, because the title's context is frequently what makes the passage comprehensible.

Semantic-only search. Embeddings are bad with product codes, invoice numbers, proper nouns and negation: "with interest" and "without interest" land close together because the texts are nearly identical.

Hybrid search (vector plus BM25) is the highest impact change there is in RAG. It almost always beats either one alone, and it is cheap to implement.

No reranking. Retrieve twenty candidates and pass them through a reranker, a smaller model trained specifically to order by relevance. It improves a lot and costs little.

No metadata. Without filtering by date, product or permission, you retrieve the revoked document from 2019. And, worse, you can retrieve a document that user had no right to see, which is a leak wearing a feature's clothes.

The diagnosis that saves weeks#

When the answer comes out wrong, look first at what was retrieved.

If the right passage was not there → it is retrieval.

If it was there and the model ignored it → it is the prompt or the model.

Two completely different situations, with completely different solutions. Without separating them, you spend months tuning prompts for a search problem.

Second: evaluation#

This is the discipline that separates product from demo, and it is the one almost nobody does.

The question is simple: can you answer, with a number, whether the change you just made improved things or made them worse?

If the answer is "I tried three questions by hand and it seemed better", you are tuning in the dark.

Evaluating AI is different from testing software. There is no deterministic pass/fail: there is a distribution of quality measured over a sample.

How to start, and it is simpler than it looks:

You need a set of cases. Twenty already changes your life. A hundred is comfortable.

Where to get them: from real logs, if you have traffic; from the cases that went wrong, which are the most valuable; and from the questions support receives. Do not invent pretty synthetic cases: they look nothing like what users actually do.

The kinds of check, cheapest to most expensive:

Deterministic. Does the answer contain the right number? Is the JSON valid? Does the id exist in the database? Whenever something can be checked with code, check it with code: it is free, fast and never wrong.

Comparison against a reference. When an expected answer exists. For free text, use semantic similarity rather than literal equality.

Model as judge. When the criterion is subjective.

Human. Expensive, and irreplaceable on a small sample to calibrate the others.

And the process: every change of prompt, model or retrieval runs against the set, and you compare before and after. It is the CI equivalent for AI.

Using an LLM as judge, without fooling yourself#

It works, with cautions that change the outcome.

Specific criterion and a small scale. "Is the answer grounded only in the provided context? Yes or no" works far better than "give it a score from 0 to 10". Continuous scores all end up at seven or eight.

Ask for the justification before the score. Same effect as reasoning in steps, and it lets you audit.

The biases are real and measured: position bias, where the judge prefers the first or last option in a comparison, which you correct by running both orders; verbosity bias, where a longer answer is judged better; and self-preference, where the model prefers text it generated itself.

And calibrate against humans. Evaluate fifty cases by hand and measure agreement. If it is low, your judge is measuring something else, and you have a number that looks objective and is not.

Third: observability#

An AI application is distributed and non-deterministic. Without traces, you cannot debug.

What to record on every call: the full prompt, including what was retrieved; the response; the model and version; the parameters; input and output token counts; latency; cost; and the correlation identifier tying it to the user's request.

In an agentic flow, record every step of the loop: which tool, with what arguments, what it returned.

Why it is non-negotiable: when a user complains about a wrong answer, the first question is "what was retrieved?". Without a trace you cannot answer, and you will not be able to reproduce it, because it is not deterministic.

The minimum dashboard per route: cost per day, p95 latency, error rate, token distribution. AI cost grows silently, and the first time most teams look is when the invoice arrives.

1. Retrievalthe quality ceilingIf the right passage did not make it into the context, no model saves you. Hybrid search is the highest impact change.
2. Evaluationmeasuring the changeTwenty real cases already change your life. Without them, every prompt change is tuning in the dark.
3. Observabilitybeing able to debugWithout a trace of what was retrieved, you cannot answer the complaint or reproduce it, because it is not deterministic.

Prompt injection: the problem with no solution#

The root cause is the same as SQL injection: instruction and data travel in the same channel. The difference is that there is no parameterised query for natural language.

Direct injection is the user sending "ignore the previous instructions". Relatively easy to mitigate.

Indirect injection is the dangerous one: the malicious content is in a document your RAG retrieves, on a page your agent reads, in an email it processes. The attacker never spoke to your system: they planted the text where the system was going to look.

What helps, and none of it is complete:

→ Clearly delimit untrusted content and instruct the model to treat it as data.

→ Least privilege on the tools. The damage depends on what the agent is able to do.

→ Human confirmation for irreversible actions.

→ Validate the output with code before acting.

→ Limit the blast radius: if the agent can only write to a draft, injection does not send email.

The mental rule: treat model output as untrusted user input.

And the point most people miss: risk does not scale with the model's intelligence. It scales with the privilege you gave it.

Teaching it to say "I do not know"#

The most valuable and least implemented behaviour.

How to get it: instruct explicitly ("if the context does not contain the information, answer that you did not find it"); give refusal examples in the few-shot, because without an example the model does not learn the shape of a refusal; require citations, because if every claim has to point at a passage, inventing gets harder; and use the retrieval score: if the best passage has low similarity, do not call the model at all.

And measure the correct refusal rate as a first class metric. The counterpoint matters: refusing too much is also bad, and it is the error that appears when you tighten too far. That is why it is a metric, not a rule.

Agents: the maths of compounding error#

If you are building an agent, one calculation worth internalising:

Ninety per cent accuracy per step, over ten steps, gives thirty five per cent accuracy at the end.

A long agent is fragile by arithmetic, not by bad implementation. The mitigations: fewer steps, deterministic verification between them, an iteration limit, and human checkpoints at the expensive points.

And on multi-agent, some honesty: in most cases, one well instrumented agent with good tools beats an orchestra. Multi-agent pays off when the subtasks are genuinely independent, or when you want different perspectives on purpose, like a critic reviewing an author.

What to do this week#

One concrete, small exercise:

Assemble twenty real questions from your domain, with the correct answer and the correct document annotated. Run your system and measure how often the right document appeared among those retrieved.

That number is the ceiling on your quality. No prompt tuning gets past it.

And you will discover, in most cases, that the problem was never the model.