RAG returns irrelevant chunks: check chunking and add the document title to each chunk
@devsingh05▲ 7 net votes
Retrieval quality often fails before the model ever sees the context. Chunks that cut sentences, or lack the document title or section heading, embed poorly. Chunk by structure (headings, paragraphs) with modest overlap, prepend the title and heading to each chunk's text, and combine keyword search (BM25) with embeddings for names, codes and exact terms. Inspect the top retrieved chunks for a few real queries before tuning the prompt.