RAG Is Mostly a Retrieval Problem

A support bot tells a customer that error ERR_2041 means their session expired and they should log in again. It says this in clean, confident prose. The actual meaning of ERR_2041 — a failed card pre-authorisation, fixable by re-entering the CVV — is in a documentation page that has been sitting in the same Postgres database the bot queries, indexed, embedded, ready. It was never retrieved.

The team's first instinct was to rewrite the system prompt. Add "do not guess." Add "if unsure, say you don't know." None of it helped, and none of it could have, because the model was not guessing about a document it had. It was answering with no relevant context at all — the one job an LLM will always do badly and always do confidently.

That gap between "the model got it wrong" and "the model never saw it" is where RAG debugging should start and almost never does.

The first question, before any prompt change

Whenever someone brings me a bad RAG answer, I ask one thing before looking at anything else: was the correct answer present in the retrieved chunks? This is a mechanical check, not a judgement call. Log what you retrieved, then read it.

// Log the retrieval, not just the answer. Without this you are guessing.
await db.query(
  `INSERT INTO rag_traces (query, chunk_ids, scores, answer)
   VALUES ($1, $2, $3, $4)`,
  [question, chunks.map(c => c.id), chunks.map(c => c.score), answer],
);

Two outcomes, two completely different projects.

If the answer was in the retrieved text and the model still got it wrong, you have a generation problem. Prompt work is now reasonable: grounding instructions, a different model, better formatting of the context block.

If the answer was not there, no prompt fixes this. You can only make the model refuse more often, which trades a wrong answer for an unhelpful one. The bug is in retrieval.

In my experience the second case dominates. It is also the less appealing one, because tuning retrieval means thinking about tokenisation and index configuration, whereas prompts feel like conversation.

Chunking is the decision you make once and live with

Before anything is embedded, something cuts the documents into pieces. That cut sets the ceiling on everything downstream — no reranker recovers information that was split away from its context.

The default that ships in every tutorial is fixed-size chunking: 500 tokens, 50 overlap, move on. It is easy and quietly destructive. The window lands wherever it lands. It splits a table from its header row, so a chunk of numbers arrives with no idea what the columns mean. It cuts a seven-step procedure after step four, so the chunk describes half a process and reads, to a model, like a complete one.

Structure-aware chunking respects the boundaries the author already put in the document. Split on headings, keep a section intact when it fits, and carry the heading path into the chunk text:

type Chunk = { text: string; headingPath: string[]; docId: string };
 
function chunkByHeading(doc: ParsedDoc, maxTokens = 900): Chunk[] {
  const out: Chunk[] = [];
 
  for (const section of doc.sections) {
    // The heading path is prepended so the embedding sees the context,
    // and so a human reading the chunk knows what it is about.
    const prefix = section.headingPath.join(' > ');
 
    if (countTokens(section.text) <= maxTokens) {
      out.push({ text: `${prefix}\n\n${section.text}`, headingPath: section.headingPath, docId: doc.id });
      continue;
    }
 
    // Only oversized sections get split, and only at paragraph boundaries.
    for (const part of splitOnParagraphs(section.text, maxTokens)) {
      out.push({ text: `${prefix}\n\n${part}`, headingPath: section.headingPath, docId: doc.id });
    }
  }
 
  return out;
}

The honest trade-off: bigger chunks carry more context and are likelier to contain a complete answer, but they dilute the embedding. A vector for 900 tokens covering three subtopics points at the average of those subtopics and matches none of them sharply. Smaller chunks retrieve precisely and then fail to say enough. I lean toward section-sized chunks and accept the precision cost, because a precise pointer to half an answer is worth less than a fuzzy pointer to a whole one. That is a preference, not a law.

Embeddings do not know what ERR_2041 is

Here is the part that surprises people who came to RAG through vector search demos. Embeddings encode semantic similarity, and ERR_2041 has no semantics. It is an arbitrary token whose meaning lives entirely in a lookup table, and its vector sits near other error-code-shaped strings — ERR_2040, ERR_3110 — because that is what it looks like, not what it means.

The same holds for product names, SKUs, version numbers, config keys, surnames. Exactly the identifiers users type when they need a specific answer.

Full-text search has no trouble with this. It matches the literal token.

-- A generated tsvector column, indexed once, maintained by Postgres.
ALTER TABLE chunks
  ADD COLUMN fts tsvector
  GENERATED ALWAYS AS (to_tsvector('simple', text)) STORED;
 
CREATE INDEX chunks_fts_idx ON chunks USING GIN (fts);
 
-- 'simple' rather than 'english': no stemming, so ERR_2041 stays ERR_2041.
-- A language-specific config would help prose recall and hurt identifiers.

That comment is a real trade-off, not a detail. english stemming improves matching on ordinary prose and mangles the exact strings you most need to match exactly. You can run both configurations; deciding which matters more for your corpus is the actual work.

Neither method is sufficient alone. Vectors handle "my payment keeps bouncing" against a page titled "Declined authorisations." Full-text handles ERR_2041. You need both.

Fusing two rankings without inventing a scale

The naive combination is to normalise both scores and take a weighted sum. This goes wrong because cosine similarity and ts_rank measure nothing comparable — weights tuned on one corpus are meaningless on the next.

Reciprocal rank fusion sidesteps this by ignoring scores entirely and using positions. Each list contributes 1 / (k + rank), so documents ranked reasonably high in both lists beat documents that top exactly one.

WITH semantic AS (
  SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> $1) AS rank
  FROM chunks
  WHERE tenant_id = $2 AND lang = $3
  ORDER BY embedding <=> $1          -- <=> is cosine distance in pgvector
  LIMIT 50
),
lexical AS (
  SELECT id, ROW_NUMBER() OVER (ORDER BY ts_rank(fts, query) DESC) AS rank
  FROM chunks, plainto_tsquery('simple', $4) AS query
  WHERE tenant_id = $2 AND lang = $3 AND fts @@ query
  ORDER BY ts_rank(fts, query) DESC
  LIMIT 50
)
SELECT COALESCE(s.id, l.id) AS id,
       -- k = 60 damps the top positions so one list cannot dominate outright.
       COALESCE(1.0 / (60 + s.rank), 0) + COALESCE(1.0 / (60 + l.rank), 0) AS rrf
FROM semantic s
FULL OUTER JOIN lexical l ON s.id = l.id
ORDER BY rrf DESC
LIMIT 30;

The cost of hybrid retrieval is honest complexity: two indexes to maintain, two ways to be misconfigured, one more query to explain when it is slow. I still consider it close to mandatory for any corpus containing identifiers, which is nearly all technical documentation.

Retrieve wide, then cut hard

Fusion gives you thirty plausible candidates. Thirty chunks is far too much context to hand a model — it buries the relevant one among distractors and pays for the privilege in tokens.

A cross-encoder reranker reads the query and each candidate together and scores actual relevance, rather than comparing two independently computed vectors. That joint encoding is why it is more accurate, and why it is slower: it cannot be precomputed or indexed.

async function retrieve(question: string, tenantId: string, lang: string) {
  // Cast wide: recall matters here, precision does not yet.
  const candidates = await hybridSearch(question, tenantId, lang, { limit: 30 });
 
  // Then cut hard. The reranker is the expensive, accurate stage.
  const scored = await reranker.score(question, candidates.map(c => c.text));
 
  return candidates
    .map((c, i) => ({ ...c, relevance: scored[i] }))
    .sort((a, b) => b.relevance - a.relevance)
    .filter(c => c.relevance > RELEVANCE_FLOOR)   // an empty result is a valid answer
    .slice(0, 5);
}

That filter line matters more than the sort. Without a floor you always return five chunks, including for questions your corpus does not cover — and five irrelevant chunks are worse than none, because they give the model material to build a confident wrong answer from. Returning nothing lets you say "I don't have documentation on that," which is true and useful.

The reranker adds latency on the critical path. Whether that is acceptable is a product decision, not an engineering one. For a support bot where the alternative is a wrong answer, it usually is.

Filtering is correctness, not tuning

In a multi-tenant system, retrieving tenant B's document for tenant A is a data breach with an LLM wrapped around it. It is not a quality issue to be improved with better ranking. The filter belongs in the query, enforced by the database, not applied to results afterwards:

-- Composite index so the tenant filter is applied via the index, not after it.
CREATE INDEX chunks_tenant_lang_idx ON chunks (tenant_id, lang);
 
CREATE INDEX chunks_embedding_idx ON chunks
  USING hnsw (embedding vector_cosine_ops);

Be aware of what this costs. An HNSW index approximates nearest neighbours across the whole table; a restrictive WHERE on top of it can leave you with far fewer than LIMIT rows, because the filter applies to candidates the index already chose. Postgres will not warn you — you just get a short result and no signal that anything was lost.

pgvector 0.8 added a direct answer to this: hnsw.iterative_scan, which lets the scan keep walking the graph until the filtered LIMIT is satisfied rather than stopping at the first candidate set. Set it to strict_order when result ordering must be exact, relaxed_order when you will rerank anyway; hnsw.max_scan_tuples bounds the work. Reach for partitioning by tenant_id when that is still not enough — with many tenants it is more predictable — but try the setting first, because it is one line.

Date and language filters are less dangerous but no less important. A bot citing a deprecated policy is confidently wrong in a way that is hard to spot from the outside.

Measure retrieval on its own

End-to-end answer quality is the number stakeholders want and the number that tells you least, because it mixes retrieval, reranking, and generation into one score that moves for reasons you cannot attribute.

Measure the retriever separately. This needs an afternoon and a spreadsheet, not infrastructure. Take thirty to fifty real user questions and note by hand which chunk actually answers each one. Then ask your pipeline a single question: did that chunk appear in the top k?

// recall@k: of the questions with a known answer chunk, how many did we surface?
function recallAtK(evalSet: EvalCase[], results: Map<string, string[]>, k: number) {
  const hits = evalSet.filter(c => results.get(c.question)!.slice(0, k).includes(c.goldChunkId));
  return hits.length / evalSet.length;
}

Run it at k = 30 before reranking and k = 5 after. Those two numbers separate two failure modes cleanly. Low recall at 30 means the candidate stage is failing and the reranker never had a chance — go fix chunking or the hybrid query. High recall at 30 but low at 5 means retrieval works and the reranker is discarding good chunks.

The value of a hand-built set is not statistical power; fifty examples prove nothing about your true accuracy. What they give you is a regression test: change the chunking strategy and you find out immediately whether you made things worse, instead of learning it from a support ticket three weeks later.

Keep the citation attached

Every chunk should travel through the pipeline with its source, and that source should reach the user.

const context = chunks
  .map((c, i) => `[${i + 1}] ${c.headingPath.join(' > ')} (${c.docUrl})\n${c.text}`)
  .join('\n\n');
// The prompt then requires each claim to carry a [n] marker.

This is partly a user-trust feature. Mostly it is an engineering one. A cited answer is a falsifiable answer: someone can click through, see that the cited page does not say what the answer claims, and file a bug that is actually actionable. Without citations you get "the bot was wrong about something last week," which is unfixable.

Citations also surface the failure I opened with. If the bot answers about ERR_2041 while citing a page on session timeouts, the retrieval bug is visible in the output rather than buried in a log nobody reads.

What I hold lightly

Chunk sizes, the k values, the RRF constant, the relevance floor — all corpus-dependent, and I would not defend any number of mine on someone else's data. They are things to measure, not inherit.

I am also unsure how long the architecture stays this shape. As context windows grow, some of what retrieval does today gets absorbed by simply passing more text. That does not eliminate retrieval — you still have to decide which of a million documents to pass, and attention over a very long context is neither free nor uniform — but it plausibly moves the boundary. If someone shows me that a large-window model with cruder retrieval beats my careful pipeline on a real eval set, I will take the simpler thing.

What I do not hold lightly is the diagnostic move at the top. Check whether the answer was in the context before changing anything else. That one is not about architecture, and it will outlive whichever architecture we are on.

The shape underneath

RAG rewards a habit that has nothing to do with AI: when a pipeline of stages produces a bad output, find out which stage lost the information before improving any of them. Teams reach for the prompt not because it is likely to work, but because the prompt is legible and fun and the retrieval layer is neither.

The model is the last stage. It can only be as right as what it was handed. Most of the time the answer was in the database all along, and everything worth fixing is in the code that failed to go get it.