Citations Are Not Decoration
Citations have become a familiar feature of AI-generated chat responses. Ask most general-purpose assistants a question today and the answer arrives with a small link, a superscript, a “Source” line underneath. It looks reassuring, and it’s often meant to be: trust me, and if you don’t, here’s where to check. In an application built over a patient record, it’s tempting to assume the same feature does the same job. The model reads the record, writes an answer, and adds a link or two pointing back at what it read. The clinician can click through if they want to. Job done.
That has a certain appeal. Conventional pages already do something similar: a dashboard tile links through to the data behind it, a summary screen links back to the data it was built from. If a link at the bottom of a page is good enough there, why would a link at the bottom of an answer need to be anything more?
A citation, on this view, is a courtesy: a nice touch at the end of an answer, rather than something the answer actually depends on.
The trouble is that showing a resource and writing about a resource are different acts. A list of a patient’s conditions, rendered by code, is definitionally exactly what the record says. There’s no room for drift, because the code doesn’t interpret anything; it reads a resource and displays it. An answer written by a language model isn’t that. The model has read across a number of resources, weighed them against a question, and produced new prose, in its own words, that didn’t exist in the record. That prose might be entirely faithful to what it read. It might also generalise two similar entries into one, or draw a conclusion the record doesn’t quite support. A link at the bottom of the page doesn’t, on its own, tell you which of those happened.
TL;DR
🔗 A citation is a claim about where an answer came from, not decoration at the end of it.
🧾 The model says what it used. Code builds the link. The two jobs stay separate.
✅ A citation is checked, not just trusted, before it reaches the page.
🕰️ A citation is pinned to the version that was retrieved, not whatever the record says by the time someone clicks.
🚪 A citation link is still a route into the record, so it carries the same permissions as everywhere else.
🎯 Verification has more than one question to answer, and they’re not the same question.
An answer is a claim, and claims need sources
There’s a moment worth borrowing from the same hospital ward as before in previous posts, though seen from a different angle this time: not the sign-in book, but the handover. When the agency member of staff hands over to the next shift, “their pain seems to have settled since Tuesday” isn’t taken on trust simply because the person saying it seems competent. A good handover names where something came from, an entry, an observation, a conversation, so that whoever’s picking up the care could go and check it themselves if they needed to.
That’s what a citation is doing. Not making an answer look credible, but giving someone else a way to verify a claim independently of the person, or the system, that made it. In an application where an answer’s fluency can easily be mistaken for its accuracy, that’s arguably more important than it sounds: an answer generated from a record should say what it’s based on, precisely enough that the claim can be checked against the record itself.
Where the link comes from
In the example application, citations are produced in two steps, kept deliberately apart. The model decides what it used. Code decides how that becomes a link.
The synthesiser agent that writes an answer, whether it’s the Patient Record Explorer, the Patient Record Enquirer, or the Patient Record Summariser, is required to return a structured object: the answer text, and a separate list of citations, each naming a resource type and a resource ID. It’s explicitly told not to put links inside the answer text itself. Identifiers below have been replaced, and the excerpt is trimmed to show the shape of the output rather than a literal response:
{
"answer": "There is a history of moderate essential hypertension, currently managed with Amlodipine and Hydrochlorothiazide.",
"citations": [
{ "resourceType": "Condition", "resourceId": "<id>" },
{ "resourceType": "MedicationRequest", "resourceId": "<id>" },
{ "resourceType": "MedicationRequest", "resourceId": "<id>" }
]
}
Turning that list into something a clinician can click is code’s job, not the model’s. A renderer agent builds a **Source** block underneath the answer, one link per cited resource, grouped by the encounter it belongs to, in the form [View {Resource Type} {date}](fhir/r4/{Resource Type}/{id}/_history/{vid}), with the version taken from what was actually retrieved rather than whatever the resource happens to say by the time the link is built. The Enquirer and the Summariser produce the same shape of output, so the same renderer agent handles all three. The application’s inline “Summarise this for me” links switch citations off, since they’re built for a different, more contained view. A document upload adds a single “View document” link instead of a list, since there’s only ever one document to point at.
There’s a reason for keeping the two jobs apart. Free text is hard to check; a model asked to “include a link” might paraphrase a resource ID, get a date wrong, or format the link differently each time. A short, structured list of {resourceType, resourceId} pairs is much easier to verify, and it’s code, not the model, that turns a verified list into something on the page.
flowchart LR
subgraph MOD[What the model decides]
direction TB
A("Synthesiser LLM") --> B("Answer text")
A --> C("Citations: resourceType and resourceId")
end
subgraph COD[What code decides]
direction TB
D("Renderer") --> E("Source block, grouped by encounter")
E --> F("fhir/r4/Type/id/_history/vid links")
end
subgraph APP[What the app decides]
direction TB
G("Link rewriter") --> H("Normal patient route")
end
C --> D
F --> G
classDef base fill:#437c9f,stroke:#2f5f79,color:#ffffff,stroke-width:1px;
classDef highlight fill:#6dc067,stroke:#437c9f,color:#1f2d1f,stroke-width:1.2px;
classDef group fill:#ffffff,stroke:#437c9f,color:#2f5f79,stroke-width:1px;
class A,B,D,E,F,G,H base;
class C highlight;
class MOD,COD,APP group;
style MOD rx:10,ry:10
style COD rx:10,ry:10
style APP rx:10,ry:10
linkStyle default stroke:#437c9f,stroke-width:1.2px;
There’s a further, smaller step in the example application: once the link reaches the browser, it’s rewritten again, from a generic FHIR reference into an application route of the form /patients/{pid}/{section}/{id}/versions/{vid}, so that clicking it opens the same patient page a clinician would otherwise navigate to, at the version that was actually cited. That rewriting is what turns a citation from a reference into an actual place someone can go, which is the subject of a later section.
Checking that the citation holds
A structured citation is easier to produce reliably than a link buried in prose, but structure alone doesn’t make it true. The example application checks a citation in two different ways, and it’s worth keeping the two apart, because they answer different questions.
Was this actually retrieved? Before a citation reaches the page, the renderer agent checks that its resource ID appears among what was actually retrieved for that question, and drops any citation that doesn’t. A second, independent check asks a validator model to run the same test: does each cited resource genuinely belong to the set the answer was built from? This is a narrow, mechanical question, closer to checking a reference exists than checking it’s the right one, and it’s cheap enough to run on every answer.
Does the source actually support the claim? This is a different, harder question, and the example application answers it with a separate validator check, run at a lower temperature, that reads the answer as a whole against the whole set of retrieved context and looks for anything the sources appear to contradict. If it finds a problem, the answer is retried with feedback, up to a handful of times, before being replaced with a fixed, honest message rather than shown as-is.
flowchart LR
subgraph GEN[Producing an answer]
direction TB
A("Retrieved resources") --> B("Synthesiser LLM")
B --> C("Answer and citations")
end
subgraph CHK[Checking it]
direction TB
D("Code: is the ID in the retrieved set?")
E("Validator LLM: membership test")
F("Validator LLM: does the source support this?")
end
C --> D
C --> E
C --> F
D --> G("Drop citation")
E --> H("Retry with feedback")
F --> H
classDef base fill:#437c9f,stroke:#2f5f79,color:#ffffff,stroke-width:1px;
classDef highlight fill:#6dc067,stroke:#437c9f,color:#1f2d1f,stroke-width:1.2px;
classDef group fill:#ffffff,stroke:#437c9f,color:#2f5f79,stroke-width:1px;
class A,B,C,D,E,F,G,H base;
class D,E,F highlight;
class GEN,CHK group;
style GEN rx:10,ry:10
style CHK rx:10,ry:10
linkStyle default stroke:#437c9f,stroke-width:1.2px;
Running both checks at the level of the whole answer, rather than sentence by sentence, is a deliberate trade-off worth naming rather than glossing over. It’s considerably cheaper, in calls and in latency, than validating each individual claim against its own citation. It also means a check passes or fails as a whole: an answer with one shaky claim among several sound ones is caught by the same mechanism as an answer that’s shaky throughout. Whether that’s the right level of granularity to check at, for a given application, is one of the questions worth asking deliberately, and it’s one we return to below.
Where the link takes you
A citation link isn’t only a pointer for a human to follow. It’s also a request, and the previous post in this series argued that asking is itself a form of access. So it’s worth asking what happens when someone actually clicks.
In the example application, a citation link opens onto the same version-history route a clinician would use to browse a record’s past versions directly, rather than a shortcut built just for citations. That route passes through the same role middleware that decides whether this user’s role can see this kind of resource, checks that the record belongs to the patient the session is scoped to before fetching the old version, and produces the same kind of audited read as any other detail view, read-only, the same as any other version. The link doesn’t open a side door into the record; it walks in through the front one, badge and all.
There’s a further detail worth being precise about: which version of the record a citation link opens. In the example application, a citation link is pinned to the specific version that was retrieved and shown to the model when the answer was written, using FHIR’s own versioned read (vread), the same mechanism the audit trail described in the first post already uses for its own entries.
That matters more for a conversation than it does for a conventional page. A clinician looking at a live record is usually shown whatever’s current, and there are established ways of dealing with it changing under them, a pop-up prompting a refresh, say. A chat answer doesn’t have that convention available to it. It’s generated from the current record at the time, in the same way the app’s default view is, but the citation underneath it might be read straight away or considerably later, and nothing about a transcript announces that the record has moved on in the meantime. Pinning the link closes that particular gap: whenever it’s followed, it shows precisely what the agent saw, rather than a record that’s since drifted just enough to make a clinician question the answer for reasons that have nothing to do with whether it was right.
The pinned version is also only ever a starting point rather than a dead end. Every detail view in the example application affords the ability to view a resource’s version history. A clinician arriving via a citation can step forward from the cited version to whatever the record says now, or browse everything in between, the same way they could from any other route into the record.
Considerations worth weighing
A few further questions tend to arise in practice.
- Membership or grounding, or both? Checking that a citation was actually retrieved is a narrow, mechanical test. Checking that the retrieved resource actually supports the claim made about it is a much harder, more judgement-laden one. They can be run by different mechanisms, at different costs, and it’s worth being clear about which one a given check is actually answering.
- Claim by claim, or the answer as a whole? Validating an entire answer against its entire context is faster and simpler than validating each sentence against its own citation, but it’s also coarser. A single check that passes doesn’t guarantee every individual claim was scrutinised with equal care.
- Why pin the citation to a version at all? A live page can flag that a record has changed while someone’s looking at it; a chat answer is read at an unpredictable point after it was generated, with no equivalent signal available. Pinning the link to the retrieved version is what closes that gap, at the cost of the extra plumbing needed to keep old versions resolvable as records keep changing.
- Does a citation link inherit permissions, or reimplement them? The previous post made the case for one access check, reused everywhere. A citation link is one more place that principle needs to hold, since it’s easy to imagine a shortcut route being added later that quietly bypasses it.
- What should a dropped citation look like to the reader? Removing a citation that fails a check is safe, but it also removes evidence the reader might otherwise have wanted, including the fact that the model reached for something that didn’t check out.
The record still has the final word
A summary is not the record, and if the two ever disagree, the record is the one that wins. That’s not a new idea, and it isn’t really specific to language models: it’s always been true of anything that aggregates, paraphrases or summarises a record on someone’s behalf. What a citation does is make that relationship checkable, by anyone, at the point they need it, rather than asking them to take the answer’s word for it.
An answer that reads well is not, by itself, evidence of anything. A citation is what turns it into something that can be checked.
Questions to ask of your own design
If you’re putting generated answers in front of clinical data, these may be worth asking:
- Does a citation identify a specific resource precisely enough to be checked, or only well enough to look reassuring?
- Is the citation produced as structured output the code can verify, or is it embedded in free text the model wrote?
- Do you check that a citation was actually retrieved, that it actually supports the claim, or both, and do you know which one each check you have is actually answering?
- When someone follows a citation link, does it carry the same permissions and patient scoping as every other route into the record?
- Does a citation point at the version of the record the model actually saw, or at whatever the record says today?
- If a citation fails a check, what does the reader see, and is that enough for them to know something was withheld?
The data used in the project to which this article relates was sourced exclusively from datasets that are known or assured to be synthetic, or have been intentionally created for illustrative purposes. The application described is a playground, built for experimentation, and has never been used in any real-life scenarios. It was created solely to demonstrate the concepts described in the article.
