Chunking and Retrieval Quality

Teams spend weeks rewording prompts and an afternoon on chunking. It is usually the wrong way round. How you split your documents decides what the search can possibly find, and no prompt recovers an answer from a passage that was never retrievable.

The reason is mechanical. A chunk is the unit that gets embedded, the unit that gets ranked, and the unit that lands in the prompt. If the sentence that answers a question is spread across two chunks, neither chunk fully answers it and neither may rank highly. The information is in your corpus and out of reach.

SOURCE PARAGRAPH
----------------
Standard-class rail travel is reimbursable when the ticket is booked seven
or more days before travel. First-class travel is reimbursable only where
the journey exceeds four hours and the employee has written approval from a
director. Receipts are required for any single item over 75 pounds.


SPLIT AT 240 CHARACTERS
-----------------------
CHUNK 118
Standard-class rail travel is reimbursable when the ticket is booked seven
or more days before travel. First-class travel is reimbursable only

CHUNK 119
where the journey exceeds four hours and the employee has written approval
from a director. Receipts are required for any single item over 75 pounds.

• Read chunk 118 on its own, as the model will. It says first-class travel is reimbursable. The two conditions that restrict it are in the next chunk.
• This is not a chunk that is merely unhelpful. It states the opposite of the policy. Retrieve it for "can I claim a first-class ticket?" and a correctly grounded, correctly cited answer will be wrong, and the citation will point at a real document.
• Chunk 119 is broken the other way. It opens with "where the journey exceeds four hours" and never says what that condition applies to. Embedded on its own it is about journey length and approvals, not about travel class, so it will not rank well for the question it half-answers.
• The split point was chosen by a character counter that has no idea a sentence was in progress. Nothing errored. Ingestion reported two chunks and moved on.
• Notice also that neither chunk carries the heading it sat under. "Travel > Rail and air > Class of travel" is the context that tells you these sentences are about rail at all.

The test to apply to every chunk is simple and worth being strict about: could a colleague who has read nothing else answer a question from this text alone, and would they know what it is about? A chunk that fails either half of that will fail in a prompt too, because the prompt gives the model exactly that much and nothing more.

import re
from dataclasses import dataclass

HEADING = re.compile(r"^(#{1,4}) +(.+)$")
MAX_CHARS = 1_200
SENTENCE_END = re.compile(r"(?<=[.!?]) +")

@dataclass
class Chunk:
    chunk_id: str
    heading_path: str
    text: str

    @property
    def text_for_embedding(self) -> str:
        # The heading trail goes into the vector, not just into the citation
        return f"{self.heading_path}\n{self.text}"

def split_markdown(body: str, document_id: str, title: str) -> list[Chunk]:
    chunks: list[Chunk] = []
    trail: list[str] = [title]
    paragraph_buffer: list[str] = []

    def flush() -> None:
        joined = "\n\n".join(paragraph_buffer).strip()
        paragraph_buffer.clear()
        if not joined:
            return
        for piece in pack_sentences(joined, MAX_CHARS):
            chunks.append(
                Chunk(
                    chunk_id=f"{document_id}#c{len(chunks):04d}",
                    heading_path=" > ".join(trail),
                    text=piece,
                )
            )

    for block in body.split("\n\n"):
        match = HEADING.match(block.strip())
        if match:
            flush()                                   # a heading always ends the current chunk
            depth = len(match.group(1))
            trail = trail[:depth] + [match.group(2).strip()]
        else:
            paragraph_buffer.append(block)

    flush()
    return chunks

def pack_sentences(text: str, limit: int) -> list[str]:
    """Fill up to limit characters, but only ever break between sentences."""
    pieces, current = [], ""
    for sentence in SENTENCE_END.split(text):
        if current and len(current) + len(sentence) + 1 > limit:
            pieces.append(current)
            current = sentence
        else:
            current = f"{current} {sentence}".strip()
    if current:
        pieces.append(current)
    return pieces

• The splitter walks the document's own structure. A heading flushes whatever was accumulating, which means a chunk never spans two sections — the boundary the author already drew is respected instead of being overridden by a character count.
• trail is maintained as a stack, so a level-three heading replaces the previous level three and keeps the level two above it. That produces "Expenses Handbook > Travel > Class of travel" rather than a flat label.
• text_for_embedding is where the heading trail earns its place. What gets embedded is the trail plus the text, so a paragraph that never says the word "travel" still lands near travel questions. The stored text stays clean for the prompt and the citation.
• pack_sentences enforces the size ceiling without ever cutting mid-sentence. It will return a chunk slightly over the limit when a single sentence exceeds it, and that is the right trade — compare it with chunk 118 above.
• The sentence pattern here is deliberately simple, and it will mishandle abbreviations and decimal figures. For policy documents with amounts like 75.00 that matters, so check its output on your own text and reach for a proper sentence splitter if the mistakes are real.
• No overlap is applied in this version. Because splits only ever land on sentence and section boundaries, there is less for overlap to rescue. Add it if your evaluation set shows answers falling between adjacent chunks, and measure rather than adding it by default.