RAG Chunking Strategy.

🐛The symptom

Your RAG replies "I don't have information about the refund window." But refunds.md literally says: "Refunds are accepted within 30 days of delivery."

🔍Step 1 — check retrieval, not the LLM

Print the top-5 retrieved chunks BEFORE generation.

Rule: if the answer isn't in the retrieved text, it's a retrieval bug, not a prompt bug. Stop tuning the system prompt.

🔍Step 2 — look at the actual chunks

Your splitter:

chunks = [text[i:i+1000] for i in range(0, len(text), 1000)]

Chunk 7 ends: "...Refunds are accepted within 30"

Chunk 8 starts: "days of delivery, provided the item..."

The fact got guillotined. Neither half embeds anywhere near the question.

🔍Step 3 — the second bug: the orphan chunk

Chunk 12 reads: "This does not apply to final-sale items."

Apply to WHAT? The "## Refunds" heading was 4000 chars earlier. The embedding has no idea what "this" is — and when it does get retrieved, it makes the model answer wrong. Worse than retrieving nothing.

🧠The concept

An embedding is ONE vector for the whole chunk. So a chunk must be (a) semantically self-contained and (b) about one thing. Fixed-size character slicing breaks both: it cuts mid-thought, and it blends two topics into an averaged vector that sits near neither.

🔧The fix, in payoff order

1. Split on structure first — headings → paragraphs → sentences. Use a recursive splitter, never raw slicing.
2. Overlap 10–15% (e.g. 800-token chunks, 100-token overlap) so boundary facts survive on both sides.
3. Prepend context before embedding:

"Returns Policy > Refunds\n\nThis does not apply to final-sale items."
Costs nothing, and usually beats upgrading to a fancier embedding model.

4. Embed small, return big: index the 200-token chunk, but feed the LLM its parent section. Retrieval wants precision; generation wants context.

📊Rules of thumb

• 200–500 tokens → precise retrieval, thin context (pair with parent expansion)
• 800–1000 tokens → safe default for prose docs
• >1500 tokens → dilutes the vector, burns context, drags in noise
• Code & tables → never split by characters. Split by function or row, or you index unrunnable fragments.

⚠️What practitioners get wrong

• Cranking top_k from 5 to 20 when the real problem is chunk boundaries.
• Changing the chunker but not re-embedding. The index still holds the OLD chunks — a chunking change means a full reindex.
• Measuring nothing. Write 20 question → expected-chunk pairs and track recall@5 before/after. One hour of work, and it turns guessing into engineering.

🎯Your exercise

Pick one doc in your RAG. Print the 5 chunks you'd need to answer a real user question. Count how many are cut mid-sentence or start with a dangling "this / it / they". That count IS your recall bug. Add heading prefixes to every chunk, reindex, and re-measure recall@5.