Practical RAG that actually works: a field guide

Almost every RAG tutorial assumes English, a clean corpus and toy questions. Then you point it at real documentation — technical specs, regulations, product sheets full of long, nested sentences — and retrieval hands back fragments that have nothing to do with the question. The model, ever obedient, hallucinates on top of them. The problem is rarely the LLM: it’s everything that happens before you call it.

This article is what I’ve learned building practical RAG systems that have to work outside the notebook — including the one that powers the AI terminal on this very portfolio: a lightweight RAG that retrieves context from my projects and experience to answer with real data, not invented claims. If you work with content in Spanish, the same friction shows up with extra force, and I’ll call those cases out explicitly.

What RAG is, and why messy real-world corpora make it harder

RAG (Retrieval-Augmented Generation) is easy to state: instead of trusting whatever the model memorized, you retrieve the relevant fragments from your corpus and pass them in as context so the model generates an answer grounded in them. Fewer hallucinations, citable sources, and knowledge you can update without retraining anything.

The hard part is the first R. Retrieving well on real documents — and especially on a morphologically rich language like Spanish — has friction the English-first ecosystem tends to ignore:

• Rich morphology. Words like “installation”, “installations”, “install”, “installed” share a root but break any naive lexical matching. A good multilingual embedding pulls them together; a tokenizer designed for English fragments them sub-optimally.
• Diacritics and normalization. “Cámara” and “camara” should collide in lexical search. If your pipeline doesn’t normalize Unicode (NFC) consistently between indexing and querying, you lose matches silently.
• Long, subordinate-heavy sentences. Many languages chain clauses far more than English does. Fixed-character-count chunking slices ideas in half far more often.

Ignoring this is the number-one reason a RAG system “almost works”.

Architecture: the four decisions that matter

A production RAG is four pieces, and each is an engineering decision — not a library you import and forget:

1. Ingestion and chunking — how you split the corpus.
2. Embeddings — which model turns text into vectors.
3. Retrieval + reranking — how you find and reorder candidates.
4. Generation with guardrails — how you stop the model from inventing what you never retrieved.

The classic mistake is pouring all your effort into step 4 (infinite prompt engineering) when 80% of the quality is decided in steps 1 and 3.

1. Chunking: the invisible work that decides everything

Chunking is where the game is won or lost. Rules I apply:

• Split by structure, not by length. Respect headings, paragraphs and lists. A fragment that starts mid-table or mid-clause is noise. Markdown and HTML give you that structure for free; a badly extracted PDF does not, so clean extraction is part of chunking.
• Moderate size with overlap. Between 300 and 600 tokens per fragment, with 10–15% overlap. The overlap keeps the answer from falling right on the boundary between two chunks and getting lost.
• Attach metadata. Source, section, date, language. Use it to filter before searching and to cite afterwards. Filtering by metadata is the cheapest optimization there is.

In this portfolio’s AI terminal the corpus is small and highly structured (my content collections of projects and experience), so each entry is already its own semantic chunk with a per-body character limit. I don’t need a sophisticated splitter: I need the total context to fit comfortably inside the window of the smallest model in the fallback chain. KISS wins when the corpus allows it — save the heavy machinery for when you genuinely need it.

2. Embeddings: FastEmbed and ONNX, on-device, no per-token bill

The embedding is what turns text into a vector where “closeness” means “similar meaning”. Two criteria dominate, and they matter doubly for non-English text: the model must be genuinely multilingual (not English with a thin coat of paint) and you must be able to run it locally and cheaply.

This is where FastEmbed on ONNX Runtime shines. Instead of depending on an embeddings API that bills per token and ships your corpus off-site, you run quantized models locally — true on-device AI:

• Zero inference cost and predictable latency: no third-party rate limits.
• Privacy by design: documents never leave your infrastructure. On projects with sensitive data — contracts, personal records — that isn’t a luxury, it’s a requirement.
• Portability: ONNX runs on a decent CPU without a GPU, and the same quantized model travels from server to edge.

It’s the same philosophy I pushed to the extreme in RecruitSecure AI, where CV embeddings are computed inside the browser with Transformers.js and Web Workers: zero data sent off-site, GDPR compliance by construction. FastEmbed/ONNX is that same idea on the server side.

To store and search those vectors I reach for Qdrant when volume justifies it: payload filtering (the metadata) combined with vector search in a single query. For small corpora, an in-memory index is more than enough.

3. Hybrid retrieval and reranking

Pure vector search fails in one very specific, very common case in technical text: exact terms. A standard code (“UNE-EN 1090”), a part reference, a proper noun. The embedding treats them as “similar” to other things and dilutes them. The fix is hybrid retrieval: combine dense search (vectors) with lexical search (BM25) and fuse the results.

Then, reranking. Initial retrieval prioritizes recall: it brings back 20–50 broad candidates. A cross-encoder reranker reorders them by actual relevance to the question, and you keep the top 3–5. It’s the difference between handing the LLM a junk drawer and giving it exactly what it needs. Retrieve broadly and then refine — it always beats retrieving narrowly and praying.

Retrieval evaluation: if you don’t measure it, you don’t have it

“It works” is not a metric. Before you touch the prompt, measure retrieval, because that’s where the quality is actually won:

• Recall@k — is the correct fragment among the k retrieved? If good context never arrives, no prompt can save it.
• MRR / nDCG — does it also arrive well-positioned? The most relevant chunk should rank at the top.
• Faithfulness — does the answer lean only on what was retrieved, or did the model fill gaps on its own?

Build a small but real evaluation set: 30–50 questions with the expected answer and the expected fragment. That’s all you need to catch regressions every time you change the chunking or the embedding model. Without it, you optimize blind and “improving” the system is a feeling, not a fact.

Hallucinations: why they happen and how to stop them

A RAG hallucination is almost always a retrieval failure in disguise: the model never received the fact, so it invented one with total confidence. My defenses, in order of impact:

1. Fix retrieval first. 70% of hallucinations vanish once the right context actually arrives. Start there, not with the prompt.
2. Instruct it to cite and to abstain. The system prompt must order the model to answer only from the context and to say “I don’t know” when the fact isn’t there. In this portfolio’s terminal the rule is explicit: “Answer ONLY with the information below. If you don’t know it, say so honestly. NEVER invent data, reasons or dates.”
3. Add a deterministic output guardrail. The prompt is necessary but not sufficient: an LLM can ignore it. So the terminal also runs a programmatic post-filter on the response, independent of the model. Belt and suspenders: the model proposes, but a layer of code has the final say.
4. Degrade gracefully. If retrieval comes back empty or a provider fails, answer “I don’t have that data” instead of improvising. My endpoint chains several providers and, if they all fall over, returns a reduced context rather than an opaque error.

That combination — prompt + deterministic filter + honest fallback — is what separates a demo from something you can put in front of a client.

The whole pattern, in one sentence

Chunk with intent, embed locally and cheaply with FastEmbed/ONNX, retrieve hybrid and reorder with a reranker, measure retrieval against a real evaluation set, and put deterministic guardrails over generation. The LLM is the last piece, not the first.

This same discipline — context engineering over real sources — is what I apply across my applied-AI work, from this portfolio’s terminal to DevFlow AI and its provider chain with fallback. The trendy model changes every quarter; the retrieval architecture around it is what lasts.

Building a RAG system and retrieval keeps letting you down?

This is exactly the kind of problem I enjoy: real-world corpora, sensitive data that can’t leave your infrastructure, and retrieval that has to be reliable, not just demoable. If you want a second opinion on your architecture or to build the system from scratch, let’s talk.