Chunking Is The New Data Modelling
Ask most engineers what chunking involves and the answer usually arrives as a single, slightly bored sentence: split the text into pieces of a sensible size, embed each one, put them in a vector store. It has the feel of a mechanical step, something to get through on the way to the interesting part. The previous post in this series described how a FHIR narrative becomes a searchable vector. It rather glossed over what happens in between: the narrative doesn’t simply become a vector. It first has to become one or more pieces of text, and something has to decide where each piece begins and ends.
Surely that decision is just plumbing.
I’m not sure it is, and this post is really an attempt to work out why. The closer we looked at chunking in the example application, the more it started to resemble a decision we’d normally take much more seriously: deciding what counts as one thing, and what counts as two. That’s not a new question. It’s the question at the heart of data modelling, and it turns out chunking is a version of it, just asked one layer later than usual, and with narrower tools for answering it.
TL;DR
✂️ Where you cut a piece of text decides what can ever be found together, not just how big the piece is.
📏 A size threshold is a boundary rule. Below it, a resource stays whole. Above it, something has to decide where the cracks go.
🧩 Splitting on structure beats splitting on length alone. Headings and paragraphs are more meaningful cut points than a fixed number of characters.
🔁 Overlap is a small insurance policy, for the meaning that happens to sit right on a boundary.
🗂️ Every chunk still carries a passport back to its source, because a fragment is only useful if it’s still traceable.
📄 Narratives and uploaded documents share one pipeline, because once something is text with a known source, it’s the same problem twice.
A cut is a modelling decision wearing overalls
Data modelling, at its heart, is perhaps a long series of decisions about what belongs together and what doesn’t. Does an address live on the patient or on a separate resource? Is a blood pressure reading one Observation with two components, or two separate observations that happen to have been taken at the same moment? FHIR has already made most of those decisions for us, which is rather the point of a standard: a Condition is a Condition, and its boundary with the world is defined by a StructureDefinition, not by whoever happens to be writing the code that day.
Chunking reopens a version of that question, but at the level of text rather than resources. A narrative generated from a Condition might be four lines long. A narrative built from a lengthy encounter summary, or an uploaded discharge letter, might run to several pages. Somewhere in between, a decision has to be made about whether that longer piece of text stays whole, or gets split, and if it’s split, where the split falls.
Get that decision right and a search for “reduced appetite” retrieves a chunk that actually contains the observation, the date it was recorded and enough surrounding context to make sense of it. Get it wrong and the same search might retrieve half an observation with no date attached, or a chunk so long that “reduced appetite” is one clause lost among a dozen others, diluting the very meaning the embedding was supposed to capture. Either way, the retrieval mechanism described in the previous post is only ever as good as the pieces it has to choose between.
That’s the sense in which chunking is modelling wearing overalls. It looks like an implementation detail. It behaves like a decision about what one unit of meaning is.
Most resources don’t need cutting at all
It’s worth saying plainly that in the example application, most FHIR narratives never reach the point of needing a decision at all. A narrative of 2,000 characters or fewer, which covers the great majority of individual resources, a Condition, an Observation, a single MedicationRequest, becomes one chunk, whole and unsplit.
That’s not a shortcut so much as the natural consequence of how these resources are shaped. A single Condition is already a coherent, bounded unit of meaning. Splitting it further wouldn’t preserve any structure worth preserving; it would just produce two smaller fragments of the same fact. Treating the whole narrative as one chunk mirrors how a clinician actually reads it: as one thing, not as a paragraph to be parsed.
flowchart LR
A(FHIR resource) --> B(Generated narrative)
B --> C{Longer than 2,000 characters?}
C -->|No| D(One chunk, whole)
C -->|Yes| E(Split into chunks)
classDef base fill:#437c9f,stroke:#2f5f79,color:#ffffff,stroke-width:1px;
classDef highlight fill:#6dc067,stroke:#437c9f,color:#1f2d1f,stroke-width:1.2px;
class A,B,E base;
class C,D highlight;
linkStyle default stroke:#437c9f,stroke-width:1.2px;
The threshold matters less for its precise value than for what it represents: an explicit line between “this stays one thing” and “this needs a further decision”. Most resource types never cross it. The interesting design work only starts on the other side.
When something has to be split
Longer text, a lengthy encounter narrative, or an uploaded document, does cross that line, and in the example application it’s split into pieces of roughly 512 tokens, estimated as characters divided by four, with a 50-token overlap between neighbouring chunks. The split itself doesn’t fall at an arbitrary character count if it can help it. It prefers to break on structure first, on markdown-style headings, falling back to paragraph boundaries when no heading is available, and only resorting to a harder cut when neither is present.
The reasoning is the same one that shapes any data model: a boundary should follow a real seam in the information, not an accident of length. A heading usually marks a genuine change of subject; a paragraph break usually marks the end of one thought and the start of another. Cutting there is far more likely to leave each resulting chunk as something that still makes sense read on its own, which is exactly the property retrieval depends on. Cutting at a fixed character count, with no regard for what’s either side of it, risks doing to a narrative what a badly normalised database schema does to a record: splitting something coherent purely because the container ran out of room.
The overlap exists for the case that structural splitting can’t fully solve: an idea that happens to straddle a boundary regardless of how sensibly it was chosen. Fifty tokens of repetition either side of a cut is a small, deliberate redundancy, insurance against losing a piece of meaning to an unlucky boundary, rather than a sign the boundary itself was wrong.
flowchart LR
A(Long narrative or document) --> B{Heading present?}
B -->|Yes| C(Split on heading)
B -->|No| D{Paragraph break present?}
D -->|Yes| E(Split on paragraph)
D -->|No| F(Split on length)
C --> G(Chunks, ~512 tokens, 50-token overlap)
E --> G
F --> G
classDef base fill:#437c9f,stroke:#2f5f79,color:#ffffff,stroke-width:1px;
classDef highlight fill:#6dc067,stroke:#437c9f,color:#1f2d1f,stroke-width:1.2px;
class A,B,C,D,E,F base;
class G highlight;
linkStyle default stroke:#437c9f,stroke-width:1.2px;
None of this removes the underlying trade-off that shows up wherever chunking is discussed: a larger chunk preserves more context at the cost of a more diluted embedding, and a smaller chunk sharpens the embedding at the cost of context. Preferring structural boundaries doesn’t dissolve that tension. It just means the decision about where to draw the line is answered by the shape of the information first, and by an arbitrary limit only when the information itself doesn’t offer a better answer.
A fragment is still a citizen of the record
Splitting a narrative into pieces would sit uneasily against everything else in this series if a chunk, once created, lost track of where it came from. The first post argued that retrieval is a read, and that every retrieved resource should be a versioned reference. The fourth extended that into the semantic layer directly: every result needs to carry enough to be traced back.
Chunking is the point where that obligation is easiest to lose, because a chunk is smaller and further removed from the resource than a whole narrative is. In the example application, each chunk row carries the patient it belongs to, the resource type, its identifier and version, the encounter it relates to and that encounter’s own date and version where one applies, an event date worked out appropriately for that resource type, the chunk’s position within the narrative it was split from, the chunk’s own text, and the vector generated from it. An uploaded document’s chunks carry the equivalent: the patient, the DocumentReference identifier and version, and the chunk’s position and creation time.
None of that is embedded into the vector itself. It travels alongside it, as metadata, which is precisely what makes it possible for a retrieved chunk to become an audited read with a versioned reference, and, further downstream, a citation a clinician can click through to the version of the record the model actually saw. A chunk that only carried a similarity score and some text would be quicker to build. It would also be a dead end the moment anyone asked where it came from.
Two kinds of text, one pipeline
A FHIR narrative isn’t the only kind of text the example application needs to make searchable. A clinical letter or a scanned report, uploaded through the document filing capability described in the first post, doesn’t arrive as a tidy narrative sitting inside a resource. It arrives as a file, with text extracted from it depending on its format, PDF, HTML or Markdown.
Once that extraction has happened, though, it’s the same problem the narratives already solve: a piece of text, of unknown length, that needs to be divided sensibly and embedded. The example application runs both through the same chunking and embedding pipeline, rather than building a second, document-specific one, so a question can be answered from whichever source actually holds the relevant information, in the same semantic space, exactly as the fourth post described.
There’s a further detail worth drawing out, because it echoes a theme from the first post in an unexpected place. When a document is filed, the summary an LLM wrote to describe it is itself embedded as one further chunk of that document, distinct from the chunks of the document’s own extracted text. That means a generated summary can be retrieved later, alongside the material it was generated from, as if it were evidence in its own right. Whether that’s the right thing to do for any given generated summary is worth thinking through carefully, in the same way Post 1 argued that generated content should always be clearly labelled as such wherever it’s kept. The point for this post is narrower: chunking doesn’t only apply to material a clinician wrote. It applies to anything that’s become text with a source, generated content included.
Why this felt familiar
None of the individual decisions described above are exotic. A size threshold, a preference for structural boundaries over arbitrary ones, a little deliberate overlap, metadata that travels with every fragment: taken separately, they read like ordinary engineering choices, the kind that get made and forgotten in a pull request.
What struck us, working through it, was how closely the shape of those decisions matched decisions we’d normally associate with modelling structured data in the first place. Deciding how big a chunk should be is not so different from deciding how granular a resource should be. Preferring a heading or a paragraph break over a fixed length is not so different from preferring a natural entity boundary over a convenient database column split. Carrying patient, encounter and version references on every chunk is not so different from the referential integrity a schema takes for granted. Chunking doesn’t feel like a new discipline invented for AI. It feels like data modelling, done in prose instead of resources, with the constraint that the units still have to be small enough for a model to search over sensibly.
That’s not a claim that chunking deserves the same formal rigour as a StructureDefinition, and this series has been careful throughout not to claim there’s one correct way to do any of this. It’s a smaller claim: that treating chunking as an afterthought, a preprocessing step to get through quickly, risks missing that the same judgement architects already bring to structured data has a role to play here too.
Considerations worth weighing
A few further questions tend to arise in practice.
- What should “size” mean? Character counts, estimated tokens and true model tokens can all disagree with one another, sometimes by a wide margin, and the choice affects how consistently a threshold behaves across different kinds of content.
- How much should structure be trusted? Headings and paragraph breaks are a reasonable proxy for meaningful boundaries, but they’re only as reliable as the text that produced them. Narrative generated from a template behaves predictably; free text extracted from an uploaded document may not.
- How much overlap is enough? More overlap catches more boundary-straddling meaning, at the cost of storing, and searching over, the same content more than once.
- What metadata does a chunk need to remain accountable? Enough to satisfy the audit and citation obligations described earlier in this series is a reasonable minimum bar, and it’s worth checking against that bar explicitly rather than assuming it’s covered.
- Should generated text be retrievable as evidence? A generated summary embedded alongside its source material can be genuinely useful to retrieve, but it’s worth being deliberate about whether, and how clearly, it’s distinguished from the material it summarises.
Questions to ask of your own design
If you’re building a chunking strategy over structured or narrative data, these may be worth asking:
- Where does your size threshold sit, and is it based on characters, estimated tokens, or true model tokens?
- Does your split prefer a genuine structural boundary, or does it fall wherever a fixed length happens to run out?
- Is there enough overlap to catch meaning that sits across a boundary, and have you checked what that overlap costs in storage and noise?
- Can every chunk your system produces be traced back to a specific resource, version and, where relevant, encounter?
- Do narrative text and other text, such as uploaded documents, go through the same pipeline, or two different ones that might quietly drift apart?
- If a generated summary is embedded alongside its sources, is it clearly distinguishable from them at the point it’s retrieved?
The model only ever sees what you decided to give it
A language model reading a chunk has no way of knowing what was cut away either side of it, or why the boundary fell where it did. It only ever sees the piece it was handed. That’s precisely why the decision about where to make the cut carries so much weight, and precisely why it deserves the same deliberate attention a data model would get, rather than being treated as the mechanical step before the interesting part begins.
It turns out the interesting part was here all along.
The data used in the project to which this article relates was sourced exclusively from datasets that are known or assured to be synthetic, or have been intentionally created for illustrative purposes. The application described is a playground, built for experimentation, and has never been used in any real-life scenarios. It was created solely to demonstrate the concepts described in the article.
