Chunking website content for retrieval

Chunking is the step where you cut content into retrievable pieces. It is usually treated as a parameter to tune, and the tuning discussion is almost always about size. Size matters less than where the cuts land.

The failure mode that size tuning cannot fix

Consider a paragraph explaining what happens when you hit a plan limit. A fixed 400 character window might cut it after “when you reach your monthly limit, the assistant”, leaving the actual consequence in the next chunk.

Now retrieval on “what happens at the limit” returns the first chunk, which contains the question's vocabulary but not its answer. The model sees a truncated sentence and either says it does not know or completes the thought itself, which is worse.

No chunk size fixes this. The problem is that the boundary ignored the structure of the text.

Split on structure first

HTML gives you a document outline for free and most chunking implementations throw it away. Headings mark topic boundaries that a human author already decided on.

A better default: split at heading boundaries, then split further only if a section exceeds your size ceiling, and when you do, split at paragraph boundaries rather than character counts.

def chunk(html_doc, max_chars=1200):
 sections = split_at_headings(html_doc) # h2, h3
 out = []
 for sec in sections:
 if len(sec.text) <= max_chars:
 out.append(sec)
 else:
 out.extend(split_at_paragraphs(sec, max_chars))
 return out

Carry the heading path into the chunk

A chunk lifted out of the middle of a document loses its context. The text might say “this is included on all paid plans” with no indication of what “this” refers to, because the answer was in a heading two levels up.

Prefixing each chunk with its heading path fixes this cheaply and improves both retrieval and the model's ability to use the chunk.

Pricing > Plan limits > Overage

When you reach your monthly credit limit, ...

This costs a few tokens per chunk and it is one of the highest return changes available.

Overlap, and why it is often overrated

Standard advice is to overlap chunks so that content near a boundary appears in both. This helps with fixed size chunking because boundaries are arbitrary.

With structural chunking, boundaries are meaningful, and overlap mostly adds duplicate content to the index. Duplicates compete with each other in retrieval and consume context budget. If you split structurally, start with no overlap and add it only if you can measure a problem it solves.

Content that resists chunking

Three types need special handling:

• Tables. Splitting a table separates rows from headers. Keep small tables whole or convert each row to a sentence.
• Lists where every item depends on the introduction. Keep the introduction with the list.
• Navigation, footers and cookie banners. Strip them before chunking or every chunk carries the same noise and similarity scores get flattened.

That last one is the most common preprocessing bug and it is invisible unless you look at your actual chunks. Print ten of them and read them. If they all start with your main navigation, that is your first fix and it is worth more than any parameter tuning.

How to know it is working

Take twenty real questions, retrieve for each, and read the top chunk. Not the answer, the chunk. If a human reading only that chunk could answer the question, retrieval is doing its job. If they could not, no amount of prompt engineering downstream will save it.

Why boundaries matter more than size

The chunking conversation is usually about size. How many tokens, how much overlap. Size matters far less than where the cuts fall.

A passage of four hundred words that contains a complete answer is useful. A passage of four hundred words that starts halfway through the setup and ends before the conclusion is not, and it is worse than useless because it will still be retrieved and the model will still answer from it.

This is why fixed character splitting underperforms so consistently. It optimises for the one property that does not matter and ignores the one that does.

Structure aware splitting in practice

Split on headings first. A section under an H2 is a unit of meaning that somebody deliberately grouped, and respecting it costs nothing.

Within a long section, split on paragraph boundaries rather than mid sentence. If a section is genuinely long, prefer several complete paragraphs per chunk over an even split.

Carry the heading path into each chunk as a prefix. A passage that begins with the document title and section heading it came from retrieves far better, because the embedding then encodes context the paragraph alone does not carry. A paragraph about limits reads very differently when the reader knows it sat under Enterprise plan.

Keep tables and lists intact. Splitting a pricing table across two chunks produces two passages that each contain half a comparison, and there is no question either can answer correctly.

Overlap, and when it stops helping

Overlapping chunks repeat some text at the boundary so a cut sentence appears whole in at least one passage. It is a reasonable hedge against bad boundaries.

It is also a way of not fixing the boundaries. Overlap increases index size, increases the chance that two near identical passages both get retrieved, and fills the prompt with duplicated text that displaces something useful.

If you are splitting on structure, modest overlap adds little. If you are splitting on character count, overlap is compensating for a decision you could simply change.

How to tell your chunking is wrong

Three symptoms, all visible without any measurement infrastructure.

Answers that restate the question. The retrieved passage contained the setup and not the resolution, so the model had nothing to work with and reflected the question back.

Answers that are correct but oddly partial, covering the first half of something and stopping. The rest was in the next chunk and it was not retrieved.

Citations pointing at a page that clearly does contain the answer, while the answer given is wrong. That combination almost always means the right page was found and the wrong passage within it was used.

Chunking content that is not prose

Most chunking advice assumes paragraphs. A real site has tables, lists, code, FAQ blocks and navigation heavy pages, and each breaks the assumption differently.

Tables should stay whole or not be indexed. Half a comparison table is not a partial answer, it is a misleading one, because the reader cannot see that a column is missing.

Lists should keep their introducing sentence. A list of five requirements retrieved without the line saying what they are requirements for is unusable, and this is one of the most common causes of an answer that is technically drawn from your content and makes no sense.

Code blocks should not be split mid block, and they usually should carry the paragraph above them, since that is where the explanation lives.

FAQ blocks chunk unusually well, one question and answer per passage, which is the one place the naive approach happens to be right.

Terms in this articleRetrieval Augmented GenerationChunkingIndex