Spent 3 weeks fighting a RAG pipeline before a guy on Discord asked why I was chunking by paragraphs

Uma Wood · 2026-09-25T07:00:04.531841+00:00

I was building a doc search tool for our internal wiki and the answers kept coming back wrong, so I blamed the embedding model for 3 weeks straight. A guy in a Discord server asked what my chunk size was and I said paragraphs, and he just replied 'try 400 tokens with overlap.' I switched to token based chunking with 50 tokens of overlap, and my answer accuracy went from unusable to about 85 percent on my 40 test questions overnight. Turns out paragraphs average 600 to 900 tokens in our docs, so the model was getting way too much noise in every chunk. I wasted close to 20 hours tuning prompts and swapping models when the real problem was upstream the whole time. Anyone else have a chunking setting that quietly wrecked their setup for way too long?

Vivisector

Loading…

Spent 3 weeks fighting a RAG pipeline before a guy on Discord asked why I was chunking by paragraphs | AI Innovations - Vivisector