Agents Are Users Too
AI-powered chat interfaces are increasingly finding their way into healthcare applications, and are beginning to appear in some of the larger patient record systems. Ask a question in plain English, get an answer, and continue the conversation. Behind the scenes might sit an AI agent: software that works out how to carry out a request and then acts on the user’s behalf, whether that means searching a record, generating an answer or carrying out a task such as booking an appointment. For the person using it, the experience may feel very different from working through a conventional application. There are no pages to navigate, no forms to complete and no obvious moment at which a record is “opened”.
That difference in feel can lead to a quiet assumption: that the usual obligations, such as auditability and secure, role-based access, somehow apply differently here. That an agent is a feature of the application rather than a user of it.
In other words, perhaps agents get a free pass.
They don’t. A chat interface is still a user interface, and an agent is another client. Whatever we expect of a web application or an API consumer, such as knowing who accessed what, who authored what and how it can be traced, we should expect of an agent too.
This article explores that idea using an example application that combines FHIR with AI agents. It is not a recommendation for how patient record systems, similar AI agent chat interfaces, and their audit and provenance, should be done. It is a set of considerations, drawn from working with the example application, that application engineers and architects may find useful when testing their own designs.
TL;DR
💬 A chat interface is still a user interface, not an exception to the rules.
📖 AuditEvent records access. Provenance records authorship. Agents need both.
🤝 Agents act on behalf of people. Record all the identities involved: the sub-agent, the practitioner and the observer.
🔎 Audit what was retrieved, not just what was displayed.
🧾 Generated output is derived data. Record what it came from.
The chat interface feels different. It isn’t.
Consider what a conventional application does when a clinician opens a patient’s record. The clinician is authenticated, the read is logged and the log says who did it, to which record and when. If the clinician updates something, the change can be attributed to them.
None of this is exotic. In health and care, being able to say who accessed a record, and who wrote what, is a baseline expectation rather than an optional extra.
Now put an agent in the middle. The clinician types a question. Behind the scenes the agent might plan a search, retrieve fragments of the record and generate an answer. From the clinician’s point of view, they asked a question. From the system’s point of view, a great deal of data was read, and something new was generated.
The obligations haven’t gone away. If anything, the interface hides more of the activity than a conventional one does, which is a good reason to be deliberate about recording it.
Two questions, two resources
FHIR already provides two resources that address different questions, and it helps to keep them apart.
AuditEventanswers who did what, to what, and when? It is the record of an interaction.Provenanceanswers where did this come from, and who is responsible for it? It is the record of authorship and origin.
An analogy from a hospital ward may help. Imagine an agency member of staff arriving to cover a shift. They sign in, and there is a record that they were on the premises and which areas they accessed. That’s the audit trail. When they make an entry in the notes, it carries their signature and the name of the clinician they were working under. That’s provenance.
We would not expect the agency worker to skip the sign-in book because they were only there for the day, or to leave entries unsigned because the ward was busy. The same reasoning applies to an agent.
Who is the agent?
The first design question is one of identity. When an agent reads or writes on behalf of a clinician, who is the actor?
There are at least three possibilities: the agent acts as the user, the agent acts as itself, or both are recorded. The first is convenient, but it makes the agent invisible. The record says a clinician did something, when in fact software did it on their behalf. The second is honest about the software but loses the person whose request set it in motion.
Before going further, a quick introduction to the cast. In the example application, a router at the front of the chat hands requests to one of several specialist sub-agents. Three of them appear in this article:
- The Patient Record Explorer (Explorer)
- The Patient Record Enquirer (Enquirer) answers common, well-defined questions by running deterministic, pre-written FHIR searches.
- The Patient Record Scheduler (Scheduler) finds and books appointment slots.
In the example application, when one of these sub-agents acts, the AuditEvent it produces names two identities: the sub-agent that acted, and the practitioner it acted for. When a clinician simply opens a record in the web interface, by contrast, the event has a single agent: the practitioner. The Device representing the sub-agent is marked as not being the initiator (requestor: false), and the practitioner as the initiator (requestor: true).
flowchart LR
subgraph INT[The interaction]
direction TB
A("Practitioner asks a question") --> B("Sub-agent")
B --> C("Retrieves FHIR resources")
end
subgraph REC[What is recorded]
direction TB
D("AuditEvent") --> E("Agent: Device (the sub-agent), not the initiator")
D --> F("Agent: Practitioner, the initiator")
D --> H("Source: Device (the application) that reported it")
D --> G("Entities: versioned references to what was retrieved")
end
C --> D
classDef base fill:#437c9f,stroke:#2f5f79,color:#ffffff,stroke-width:1px;
classDef highlight fill:#6dc067,stroke:#437c9f,color:#1f2d1f,stroke-width:1.2px;
classDef group fill:#ffffff,stroke:#437c9f,color:#2f5f79,stroke-width:1px;
class A,B,C,D,E,F,H base;
class G highlight;
class INT,REC group;
style INT rx:10,ry:10
style REC rx:10,ry:10
linkStyle default stroke:#437c9f,stroke-width:1.2px;
So, what does this look like in practice? The excerpt below is abridged from an AuditEvent written by the Explorer. Identifiers and the practitioner’s name have been replaced, and the comments have been added for this article, so it illustrates the shape of the resource rather than being valid JSON as it stands.
{
"resourceType": "AuditEvent",
"action": "R", // a read interaction
"outcome": "0", // success
"agent": [
{
"who": {
"reference": "Device/<id>", // the sub-agent that acted
"display": "Patient Record Explorer Agent"
},
"type": { "coding": [ { "code": "AGENT" } ] },
"requestor": false // not the initiator
},
{
"who": {
"reference": "Practitioner/<id>", // the person the sub-agent acted for
"display": "<practitioner name>"
},
"requestor": true // the initiator
}
],
"source": {
"observer": {
"reference": "Device/<id>", // the application that reported the event
"display": "AI Agent Service"
}
}
// ...remaining elements omitted
}
Each identity is a reference to a resource, with a display name so that the record can be read without a lookup (a FHIR search _include parameter or separate read requests). There are three in all: the practitioner who asked, the sub-agent that acted , and the observer, which is the application that reported the event. On a conventional read in the web interface, the source names the application client instead, and there is no sub-agent to record.
The same pattern appears in Provenance. When the Scheduler booked an appointment, the Provenance resource recorded the Scheduler’s Device as the agent, acting on behalf of the practitioner. This is exactly what Provenance.agent.onBehalfOf is for: it captures both the software that acted and the person for whom it acted. An abridged example follows in the section on generated output.
How much granularity to give an agent’s identity is a design decision, and it starts with what counts as an agent. A workflow made up of sub-agents might be treated as one entire agent, or as a group of individual agents. Two approaches are:
- One identity per sub-agent. The record shows which sub-agent acted, so you can tell a search agent from a scheduling agent. It means more identities to manage.
- One identity for the whole workflow. The record shows that the agent workflow acted on the clinician’s behalf, with a single identity to manage. Questions such as “which agent did this?” become harder to answer later.
The example application records at the sub-agent level, as the excerpt above shows. Which approach fits your own design is for architects, engineers, governance bodies and audit requirements to decide together.
How much granularity to give an agent’s identity is a design decision, and it starts with what counts as an agent. A workflow made up of sub-agents might be treated as one entire agent, or as a group of individual agents. Two approaches are:
- One identity per sub-agent. The record shows which sub-agent acted, so you can tell a search agent from a scheduling agent. It means more identities to manage.
- One identity for the whole workflow. The record shows that the agent workflow acted on the clinician’s behalf, with a single identity to manage. Questions such as “which agent did this?” become harder to answer later.
The example application records at the sub-agent level, as the excerpt above shows. Which approach fits your own design is for architects, engineers, governance bodies and audit requirements to decide together.
Retrieval is a read
When an agent retrieves information to answer a question, that is a read, and in many respects it is the most important one. A semantic search hit that is passed to a language model is effectively a disclosure of that information to the process assembling the answer, whether or not any of it ends up on the screen.
This suggests a principle: audit what was retrieved, not only what was displayed.
Back on the hospital ward, it is the difference between a sign-in book that records that the agency member of staff was on the premises and one that also records which sets of notes they took from the trolley, including any they read but never mentioned to anyone.
In the example application, the Explorer wrote an audit event for every retrieval, listing each resource that came back. The excerpt below is abridged and redacted in the same way as before. It comes from a question about whether a synthetic patient record contained any history of hypertension.
{
"resourceType": "AuditEvent",
"type": { "code": "rest" },
"subtype": [ { "code": "search-type" } ], // recorded like any other FHIR search
"action": "R",
"entity": [
{
"role": { "code": "24", "display": "Query" },
// what the user typed (base64): "any history of hypertension?"
"query": "YW55IGhpc3Rvcnkgb2YgaHlwZXJ0ZW5zaW9uPw==",
"detail": [
{
//what the model searched for (base64): 'patient's history of hypertension, high blood pressure, or hypertensive disease'
"type": "llm-generated-semantic-query",
"valueBase64Binary": "cGF0aWVudCdzIGhpc3Rvcnkgb2YgaHlwZXJ0ZW5zaW9uLCBoaWdoIGJsb29kIHByZXNzdXJlLCBvciBoeXBlcnRlbnNpdmUgZGlzZWFzZQ=="
}
]
},
// one entry per retrieved resource, each pinned to a version
// (three of fifteen shown)
{ "what": { "reference": "DocumentReference/<id>/_history/1" } },
{ "what": { "reference": "Condition/<id>/_history/1" } },
{ "what": { "reference": "Encounter/<id>/_history/1" } }
// ...twelve more in this example
]
}
Three details are worth drawing out.
The retrieval is recorded as a read. It uses the same vocabulary as any other FHIR search: a REST interaction, subtype search-type, action R. Nothing new had to be invented to describe what the agent did.
Each retrieved resource is a versioned reference. A reference ending in _history/1 identifies a specific version rather than whatever the resource says today, so the audit trail identifies the version that existed when the retrieval was recorded. That matters when records change, because the audit trail needs to remain accurate about which version of a record was involved.
The user’s question and the search the model runs are not the same thing. A language model will often rewrite a question before searching. The user’s own words are stored as the query, and the model’s rewritten search text is recorded separately, in a detail entry clearly labelled as generated. FHIR carries both values as base64, which is why the excerpt shows them encoded. Because the data is synthetic, we have left the encoded values in place, so you can paste them into any base64 decoder and see for yourself. Decoded, they read:
- What the user typed (
entity.query): any history of hypertension? - What the model searched for on their behalf (the detail entry labelled
llm-generated-semantic-query): patient’s history of hypertension, high blood pressure, or hypertensive disease
The person asked a short question, and the model widened it to cover related terms. The audit record reflects what the person asked, while still showing what the system did with it.
Not every agent needs a model to write its search, and the record reflects that. The Enquirer works from a library of pre-written FHIR queries, each linked to a range of natural-language phrasings of the question it answers. When a question arrives from the router, the Enquirer converts it into an embedding, a numerical representation of its meaning, and looks for the pre-written query whose phrasings sit closest to it. If one is close enough, that query is run as written, with the patient (and any dates) filled in by code; if none is, the Enquirer says it can’t help rather than guessing. No language model writes the search.
Its audit event has the same shape: the user’s own words are the query, and the search that actually ran is recorded in a separate detail entry, labelled resolved-fhir-search. For the question “Next of kin?”, the two read:
- What the user typed (
entity.query): Next of kin? - What ran (the detail entry labelled
resolved-fhir-search):RelatedPerson?patient=Patient/<id>&relationship=http://terminology.hl7.org/CodeSystem/v2-0131|N
The N is the next-of-kin relationship code, and the single RelatedPerson that came back appears as one versioned reference. The record shows both what was asked and exactly what was run.
The Scheduler’s tool calls were audited in the same way, one event per call. Its search for available slots, for example, recorded the FHIR search that was run as the query, and the slots and schedule it returned as versioned references.
There are costs to consider. For the Explorer, version-pinning meant an extra read for each retrieved resource, and auditing at this level of detail produces large audit events. Whether the volume is acceptable depends on the use case, but it is better to make that choice deliberately than to discover later that retrieval was invisible.
Generated output is derived data
The other side of the coin is what agents produce. A generated summary is new information derived from existing records. If someone later asks where a statement came from, we should be able to answer.
In the example application, the Summariser’s generated summaries were recorded alongside references to the source records they were derived from, with the generated text explicitly typed as model output. For bookings, the Scheduler’s Provenance resource pointed at the specific version of the appointment created, with the Slot it was booked from recorded as a source, also at a specific version.
Here is the booking first. The excerpt below is abridged from the Provenance the Scheduler wrote when it created an appointment. Identifiers and the practitioner’s name have been replaced, and the comments have been added for this article.
{
"resourceType": "Provenance",
"target": [
{ "reference": "Appointment/<id>/_history/1" } // the appointment that was created, at a specific version
],
"activity": { "coding": [ { "code": "CREATE" } ] }, // what was done
"agent": [
{
"who": {
"reference": "Device/<id>", // the sub-agent that acted
"display": "Patient Record Appointment Scheduler Agent"
},
"onBehalfOf": {
"reference": "Practitioner/<id>", // the person it acted for
"display": "<practitioner name>"
}
}
],
"entity": [
{
"role": "source", // what it was derived from
"what": { "reference": "Slot/<id>/_history/2" } // the slot it was booked from, at a specific version
}
]
// ...remaining elements omitted
}
The derivation is all there: what was created, by which sub-agent, on whose behalf, and from which version of the slot.
The Summariser’s output is recorded differently, inside an AuditEvent, next to references to the records it was derived from. This excerpt is abridged in the same way, and the summary itself is truncated. In the real event it runs to several paragraphs.
{
"resourceType": "AuditEvent",
"action": "R",
"agent": [
{
"who": {
"reference": "Device/<id>", // the sub-agent that acted
"display": "Patient Record Summariser Agent"
},
"requestor": false
},
{
"who": {
"reference": "Practitioner/<id>", // the person it acted for
"display": "<practitioner name>"
},
"requestor": true
}
],
"source": {
"observer": {
"reference": "Device/<id>", // the application that reported the event
"display": "Application Client"
}
},
"entity": [
{
"role": { "code": "24", "display": "Query" }, // the versioned Questionnaire used to gather the source records (base64)
"query": "UXVlc3Rpb25uYWlyZS9wYXRpZW50LXN1bW1hcnktdmlldy9faGlzdG9yeS8x",
"detail": [
{
"type": "llm-generated-summary", // labelled as generated
// the summary itself (base64, truncated)
"valueBase64Binary": "TXJzIFNhdXJhIFN3YW5pbWF0aGFuIGlzIGEgZmVtYWxlIHBhdGllbnQgb2YgQmFuZ2xhZGVzaGkgZGVzY2VudCwgYm9ybiBpbiAxOTQ5LiBTaGUgaXMgbWFycmllZCBhbmQgc3BlYWtzIEJlbmdhbGksIHJlcXVpcmluZyBhbiBpbnRlcnByZXRlciBmb3IgaGVhbHRoY2FyZSBpbnRlcmFjdGlvbnMu"
}
]
},
// the source records it was derived from, each pinned to a version (three of eighty-four shown)
{ "what": { "reference": "Patient/<id>/_history/1" } },
{ "what": { "reference": "RelatedPerson/<id>/_history/1" } },
{ "what": { "reference": "MedicationRequest/<id>/_history/1" } }
// ...eighty-one more in this example
]
}
As before, the base64 is left in place so you can decode it yourself. The query decodes to Questionnaire/patient-summary-view/_history/1, and the start of the summary reads:
Mrs Saura Swanimathan is a female patient of Bangladeshi descent, born in 1949. She is married and speaks Bengali, requiring an interpreter for healthcare interactions.
Two details carry the weight. The generated text is labelled llm-generated-summary, so it can be told apart from something a clinician wrote. And every source record is a versioned reference, so the summary can be traced back to what it was built from.
Whether that trace lives in an AuditEvent or a Provenance resource is a design choice. Provenance is purpose-built for derivation and is arguably the more natural home, whilst an AuditEvent is a convenient place to keep everything about one interaction together. The example application used both: an AuditEvent for summaries and a Provenance for bookings. The principle matters more than the container: generated content should carry a record of what it came from and who, or what, produced it.
Considerations worth weighing
A few further questions tend to arise in practice.
- Where does audit live? If every tool has to remember to record its own activity, coverage depends on discipline. If it is a property of the platform, coverage comes for free. Either can work, but they fail differently.
- How much detail is enough? Recording every retrieved resource is thorough, but it is also heavy. Recording only what was displayed is light, but it misses what the model saw.
- Whose identity travels with the call? This matters for audit, but it matters even more for permissions, which is where we go next.
- Audit records are data too. They may contain a clinician’s question or a generated summary. Treat the audit store with the same care as the record it describes.
Questions to ask of your own design
If you’re putting an agent in front of clinical data, these may be worth asking:
- Have you decided what counts as an agent for audit purposes, and does the record show who it acted for?
- Is a retrieval recorded as a read, including records the user never saw on screen?
- Can you tell what the user actually asked from what the system searched for, and whether that search was written by a model or came from a pre-written query?
- Do references point at a specific version of a record, or whatever it says today?
- Does generated content record what it was derived from?
- Is audit a property of the platform, or something each tool must remember to do?
- If someone asked “why did the system say this?” six months from now, could you answer?
An agent doesn’t get a free pass
None of this is new. Audit, provenance and attribution are the same questions application architects have been asking for years. What changes is that the interface makes them easy to forget.
The interface may be a conversation, but the obligations are the same as they are for any other client. An agent doesn’t get a free pass. It gets an identity.
The data used in the project to which this article relates was sourced exclusively from datasets that are known or assured to be synthetic, or have been intentionally created for illustrative purposes. The application described is a playground, built for experimentation, and has never been used in any real-life scenarios. It was created solely to demonstrate the concepts described in the article.
