What does good data architecture in healthcare actually look like?

Table of Contents

1. What makes healthcare data architecture "good"?
2. This is the technical half of a two-part series
3. Where does a generic lakehouse lose the clinical fact?
4. The six properties of good healthcare data architecture
5. How should bronze, silver, and gold work in a healthcare lakehouse?
6. What does complete HL7, FHIR, and claims ingestion require?
7. How does silver handle identity, quality, and clinical semantics?
8. What does PHI governance look like when you can demonstrate it?
9. How should AI consume a healthcare data platform?
10. What are the most common ways a healthcare data platform fails?
11. Ten questions to audit your healthcare data platform

What makes healthcare data architecture "good"?

Good healthcare data architecture is a clinical data platform a CTO can defend on a Monday morning, in front of whoever is asking why a number moved. Running on a lakehouse doesn't qualify it, and neither does a folder layout called bronze, silver, and gold. What qualifies it is six properties holding on the same data at the same time:

1. Clinical meaning survives every hop from source to consumer.
2. Patient and encounter identity is modeled explicitly.
3. Any bad row can be traced back to the payload and the transform that produced it.
4. PHI classification drives access, retention, and audit.
5. Product, BI, and AI all read from one governed platform.
6. The platform keeps running after a feed breaks.

We build to that bar on Databricks, but the properties are what define it. You could meet them on another stack, and you can miss them on this one.

We start with where a typical lakehouse quietly loses clinical facts, then walk through each property with a test you can run against your own platform. The second half covers layer design, HL7, FHIR, and claims ingestion, governance, AI, five common failure patterns, and a ten-question self-audit to bring to your next architecture review.

This is the technical half of a two-part series

On 24 September, Bruno Ferrari published the business case: Before AI, data: why we're building a Data Engineering practice at Light-it. That piece explains why we built the practice. This one sets out the architecture standard we hold it to.

Where does a generic lakehouse lose the clinical fact?

A medallion layout can have exactly the right shape and still drop the detail a clinician would recognize. The jobs stay green while the context that made a value true disappears somewhere between bronze and gold.

Consider a composite case, assembled from recurring patterns rather than taken from one client. A prior-authorization model went live with sign-off from clinical operations. Three weeks later, clinicians had stopped opening its suggestions. Nobody had touched the weights, the threshold, or the prompt. The problem sat upstream, in the nightly job that fed the model.

That job received two records for the same stay: an HL7 v2 MDM message carrying the document type in TXA-2 as a local code, and a FHIR DocumentReference typed with LOINC (18842-5 for a discharge summary, 11506-3 for a progress note). It treated the pair as one document whenever patient_id and encounter_id matched, a key that groups a hospital stay rather than a document. After matching, it cast both types to string, dropped the code systems, and kept whichever text was longer. Because note_text was populated, the null check passed. No row count could have caught a missing code system.

The same job had also collapsed identity. Two MRNs from two assigning authorities had been forced into a single patient_id. When the source later sent a merge, the surviving MRN arrived in PID-3 and the retired one in MRG-1 with its assigning authority. The pipeline kept PID-3 and discarded MRG-1, both authorities, and any crosswalk, so the surviving MRN looked clean and the retired one could no longer be traced to it.

When the team tried to reconstruct what had happened, there was nothing to replay. The first object the pipeline stored was already note_text. The MSH-10 control id, the DocumentReference id and version, and the transform that dropped TXA-2 were all gone, which meant lineage ended at a job name.

That one job shows three of the five ways clinical context usually goes missing. Each of them comes down to a specific field that never reached silver.

A code stored without its code system

A local observation code, a LOINC code, and an ICD-10-CM code can all land in the same string column, at which point a lab code and a diagnosis look identical. Units from OBX-6 or Observation.valueQuantity tend to disappear on the same hop.

One timestamp standing in for several clocks

A column called event_ts usually holds ingest time. Message time (MSH-7), observation time (OBX-14), FHIR effective versus issued, and the moment a document was signed are separate facts. Keeping one of them is a modeling decision. Dropping the rest loses information, and you end up with a timestamp nobody can name.

An identifier without its assigning authority

Each PID-3 entry carries a value and an assigning authority. Keep the digits, drop the authority, and two patients at two facilities become one person whenever their numbers happen to match.

A bronze layer that starts after parsing

If the first object you store is a CSV an integration engine has already flattened, TXA, PID, and OBX are gone before your platform ever sees them. Bronze should hold the payload the sender transmitted, its envelope ids, and a quarantine path for bytes that didn't parse. If you can't replay the original message, what you have isn't bronze.

A grain chosen before the fact has a name

When one encounter_id sometimes holds a FHIR Encounter, sometimes a scheduling slot, and sometimes an 837 claim line, it answers three different questions with one number. The dashboard looks stable, while finance, clinical operations, and the model are each right about a different fact.

Moving to a new platform won't bring back a dropped code system. The six properties below are how you check whether the context is still there.

The six properties of good healthcare data architecture

Treat these as acceptance criteria. Each comes with a Monday test, and a property counts as met only when its test passes. Later sections cover how to build for them.

1. Clinical meaning survives

A clinician and an engineer should be able to point at the same fact and agree on what it is. That requires the code, the code system, and the display to travel together, whether the system is LOINC, SNOMED CT, ICD-10-CM, CPT, HCPCS, NDC, RxNorm, or whatever local system the source actually sent. Observations keep their units, reference range, and interpretation. Documents keep their type, author, authenticator, and confidentiality. Status survives too (preliminary or final, order or result, planned or completed), along with both clinical effective time and recorded time. A local code with no standard mapping stays in the table, flagged as unmapped.

Monday test: Pick one lab result and one discharge medication from a gold table, and read the code, code system, and unit from silver without opening the source system. Then do the same for one document's type and either its status or its signed time.

2. Identity is explicit

An MRN is an identifier issued by an assigning authority, so store its type, system, and value. Apply the same pattern to encounters, appointments, claims, coverage, practitioners (NPI plus local id), organizations, and locations.

The crosswalk is a table in its own right. Every row records the match method (deterministic or probabilistic), a confidence score, the decision, and when it was made. Probabilistic matches wait in a review queue until someone accepts them. Merges and unmerges are stored as events, so both identifiers survive along with the interval during which they were linked.

Monday test: Take two source identifiers and say whether they belong to the same person, who decided that, and which one was retired. Then apply the unmerge the source sent. Both identifiers should resolve independently again, with the linked interval still on record. If your source system can undo a merge and your warehouse can't, the warehouse is the one that's wrong.

3. Lineage is reconstructable

A job name tells you where to start looking, while lineage tells you the answer. For any row you need its full set of parents: source system, message or resource id and version, file or interchange id, and transform version. Most rows have more than one. Corrections and voids arrive as new facts linked to the original, and the link itself carries the mapping version, meaning which field, which code-system map, and which survivorship rule applied.

Monday test: Someone flags a wrong row. Starting from the row alone, open its bronze payloads and the transform version that built it. If the answer lives in a Slack thread, you don't have lineage yet.

4. PHI is the control plane

Who can see a patient should be a rule attached to the data, rather than a filter someone has to remember to add to each dashboard. Classify every record at ingest as PHI, limited data set, de-identified, or non-PHI operational. Attach a purpose of use to each grant (treatment, payment, operations, or research), and keep in mind that model training doesn't count as operations by default. Column and row rules follow from the class.

Your instrumented audit log should answer who saw which patient, when, through which consumer, and for what purpose. Retention follows the data class and your agreement with the source. A de-identified dataset names its method (Safe Harbor or expert determination) and the snapshot it came from. Break-glass access requires a reason and gets reviewed afterward.

Be precise about what a revoke does: it blocks new reads on the paths you've wired, and anything that already left the platform is outside its reach.

Monday test: Remove one group's grant, then confirm that a new warehouse query, a new product API call, and a new export all fail to return the column. Finish by listing every copy already outside the platform. A revoke that stops a notebook but not the API means you are running a second, quieter platform alongside the first.

5. One platform serves every consumer

Product, BI, and AI all read governed tables, or projections that name their parent table. Every metric has an owner, a grain, and filters defined in code. "Active patient" is a good example: it lives in gold, defined once, with a date.

Different consumers can legitimately use different grains. A claim-line mart and an encounter API may disagree, provided each grain is written down and the gap between them is explained. A documented grain without that explanation is still a fork.

Monday test: Take the number in the board pack, the number in the product, and the number in the eval set. Show one query that produces all three, or a documented derivation that names each grain and explains why they differ. Three numbers that differ for stated reasons are fine. Three that differ by accident are how a board stops trusting its data team.

6. It still works on Monday

Replay should be a routine operation, and that depends on idempotency keys scoped to the sender, because message control ids collide across facilities. FHIR identity combines source, resource id, and version when the source provides one. X12 interchanges use the sender's control numbers. Confirm the exact composite key feed by feed. Replaying the same input must produce the same keys and never a duplicate clinical fact.

Unparseable bytes go to quarantine with the error attached. A surprising clinical value is a different case entirely, so flag it and keep it. Contract tests fail the build whenever a document-type set or profile you depend on changes.

Monday test: Replay yesterday's ADT feed from bronze. Repeated messages shouldn't create a second encounter, and a later correction should update the current value while preserving the prior one. Afterward, reconcile messages read, rows quarantined, silver keys, and gold keys against the first run, without anyone editing a row by hand. If replay depends on the one engineer who knows the tricks, the platform doesn't work on Monday yet.

How should bronze, silver, and gold work in a healthcare lakehouse?

The medallion pattern is a filing system. What turns it into healthcare architecture is the rule for what each drawer may hold: bronze is what arrived, silver is what the fact means, and gold is the contract a consumer depends on.

Access, retention, audit, and lineage apply at every layer. A clinical payload in bronze is already PHI, so gold inherits those obligations rather than creating them.

Here is how three common facts move through those layers: a clinical document, a patient, and a claim.

Silver keeps the four medication facts (request, dispense, administration, and statement) separate, and a gold contract combines them only when it says which ones it uses. Teams sometimes leave bronze ungoverned on the grounds that it's "just raw." In our experience, that is usually where the incident starts.

What does complete HL7, FHIR, and claims ingestion require?

In practice, interoperability means the set of messages you can replay. Ingest the transactions your consumers actually need, and ingest each of them in full.

HL7 v2. Keep ADT for identity and encounters, merge history included. Keep orders as ORM or OML, results as ORU with OBR and OBX still structured, and documents as MDM with TXA intact. Ingest DFT only when charges have a real consumer. Store MSH-10 and the sending facility on every row, since a control id is unique per sender and nowhere else.

FHIR R4. Keep the resource id, meta.versionId when the source sends it, and identifiers with their system. US Core constrains a profile, but conforming to it is no reason to drop the remaining fields. For every field promoted to silver, record the source element and the mapping version; when one resource doesn't map cleanly to one row, a documented derivation covers it. For DocumentReference, include the type coding and whichever content element the source populated. A FHIR server search is a query, so it won't give you a stored payload to replay.

X12. Keep the interchange envelope so any claim can be replayed. Map the transactions you actually receive: 837 for institutional, professional, or dental claims, 835 for remittance, 270/271 for eligibility, and 278 when prior authorization has a consumer. Treat replacements and voids as new facts linked to the original claim. Loop and element names follow the implementation guide you work from.

NCPDP. Keep prescription messages, plus pharmacy-claim transactions when a pharmacy-benefit feed is in scope. You'll see both NDC and RxNorm. Keep both, because a package code and a clinical drug code answer different questions. Message names follow your implementation guide.

C-CDA. Keep the document. The narrative and the structured entries are both facts, and section codes and template ids travel with each entry. Neither one replaces the other, and neither replaces the signed document.

Every map into silver is versioned, with one transform per message type, resource, or transaction set. A CI fixture feeds in a known payload and asserts the resulting silver row, including the code system and the identifier authority. Ingestion you can't replay gives you a shadow of the standard rather than the standard itself.

How does silver handle identity, quality, and clinical semantics?

The six properties describe what has to be true. This section covers how Silver makes those tests pass.

Identity

Match on MRN plus assigning authority, member id plus payer, and NPI before running any probabilistic score, and hold probabilistic matches in a queue until a decision is recorded. A merge event stores the surviving id, the retired id, both authorities, and the start of the link. An unmerge closes that interval and leaves both ids addressable. When names or addresses conflict, a written survivorship rule selects the value and keeps the rejected one on the row, and overlays keep both identities. Authors, organizations, and locations follow the same pattern, because a free-text author field won't hold up in an audit.

Quality

Four checks form the baseline for any feed and its contract. Some feeds need more.

• Structural: the message parsed.
• Terminology: the code exists in the system it claims.
• Temporal: result time sits in a plausible relation to order time, or an explicit exception explains why it doesn't.
• Referential: the observation points at an encounter you hold, or it's marked external.

Reconciliation should be readable by a team lead without help: messages received, messages that reached silver, source identifiers, persons on the crosswalk, and orphan results. Freshness is a per-feed SLO set on the gold contract that depends on that feed, because a perfect table from yesterday still puts the wrong chart in front of someone today.

Clinical semantics

A medication is four facts. MedicationRequest comes from the order, MedicationDispense from the fill, MedicationAdministration from the MAR or an administration message, and MedicationStatement from what the patient or chart says is in use. Silver keeps them apart, and gold combines them only when it names which ones.

A condition is three facts: the problem-list entry, the encounter diagnosis, and the claim diagnosis. None of them overwrites the others, and claim sequence stays a billing fact. Observations, diagnostic reports, and documents also stay separate. A report can reference observations, but it shouldn't be flattened into a bag of text columns.

The map from local to standard codes is versioned, and unmapped codes stay visible. Lineage is many-to-many: a silver document lists each bronze parent with the mapping version on that link, and a late correction writes a new version while keeping the prior value.

What does PHI governance look like when you can demonstrate it?

Your on-call engineer and your auditor need to read the same controls. A quarterly governance council and a spreadsheet of data owners won't satisfy either of them.

What to have in place

• Purpose-of-use grants for treatment, payment, operations, and research, with model training called out separately
• Fixture tests for every map that touches a code system or an identifier
• Contract tests that fail CI when a feed's type set or profile changes
• A named owner on each gold table, stored in the same repo as the table
• Quarantine counts and freshness available as queries
• Grants defined in code next to the tables, so a manual console change shows up as drift
• An access record that answers who, which patient, when, for what purpose, and through which consumer
• An export rule: the copy stays in governed scope, or the export is blocked
• Retention for both clinical rows and access records, on clocks you can show
• Break-glass access with a stated reason and a later review

What you can demonstrate

Pick a principal whose grant you removed and show that a new query, a new API call, and a new export all come back without the column. Run the access query for a known patient and show the events your warehouse records. Break a code-system fixture and watch CI fail. That sequence is what enforcement looks like.

What you shouldn't claim

A revoke won't reach a file copied out last week, a BI extract sitting in a cache, or a dataframe someone saved from a notebook. Audit coverage extends only to the consumers you've wired to the log, so name them. On Databricks, we enforce the controls we've wired through catalog grants and the platform audit log, and we're explicit with clients that this doesn't cover copies that already exist.

A control you can't demonstrate on a revoked user and a known patient is still a policy. Policies have value, but they aren't enforcement yet.

How should AI consume a healthcare data platform?

Treat a model as one more reader. It gets a contract, a snapshot, and a purpose of use, and never a private copy of the source system.

A training set is a snapshot defined by its as-of time, population, label definition in code, PHI class, exclusion rules, and transform version, which means someone else can rebuild it. Features at prediction time include only what was knowable at that moment; otherwise your eval numbers won't match what happens on the ward. Online features match the snapshot, or the difference is captured in a versioned contract.

In the prior-auth example, the packet lists its fields explicitly, and that list defines the minimum necessary. If you run a vector index, it indexes a governed text field and carries the document key and the PHI class. ClinicFrame is one product consuming these contracts, and Bruno's piece covers the product side.

What are the most common ways a healthcare data platform fails?

Each of these five patterns breaks at least one of the six properties, and each has a detection test you can run this week.

PHI swamp

One schema, broad grants, classification deferred, and de-identification done on someone's laptop. This breaks properties 4 and 6. Detection: list every principal and every export path for a single PHI column. If you can't finish the list, you're in the swamp.

Metric fork

Three visit counts (claim lines, encounters, and last quarter's eval extract) that differ with no explanation. This breaks property 5. Differences that are documented and explained are fine, but a written grain with no explanation still fails. Detection: ask product, finance, and the eval owner why their grains differ. Silence, or an answer nobody can source, means you have a fork.

FHIR façade

A relational model behind a FHIR API can be a sound product boundary. It fails when an API field can't be traced to a source element and a mapping version, or to a documented derivation. A field with neither breaks property 1. Detection: pick one field and show that pair. If you can't, the façade made it up.

AI sidecar

The model reads from a bucket or vendor store loaded by a one-off export, with no path back to bronze and no shared grants. This breaks properties 3, 4, and 5. Detection: trace one prediction to its snapshot id and bronze parents. If the trail ends at a bucket, it's a sidecar.

Warehouse as EHR backup

Vendor tables copied column for column, internal ids and all, while facts that only ever arrive as messages never become tables. This breaks properties 1 and 2, and it doesn't work as an EHR recovery plan either. Detection: name one fact that arrives only as a message. If bronze contains nothing but vendor tables, you copied a schema.

Ten questions to audit your healthcare data platform

Run these against a single feed that's already in production. They restate the six properties, plus ingestion, as questions.

1. Take one lab result in a gold table. Can you name its code, code system, units, and source message or resource id?
2. Is patient identity modeled as an assigning authority plus an identifier, with merges and unmerges kept as events, or is it a single patient_id column?
3. When someone flags a wrong row, can you walk it back to its bronze payloads and transform version without asking anyone?
4. If you revoke one group's access, do new reads fail on every path you've wired (warehouse query, product API, and new export), and can you list the older copies the revoke won't reach?
5. Do the product API, the metric mart, and the eval snapshot share one encounter grain? If not, does each publish its grain and explain the difference? An undocumented or unexplained difference fails.
6. Can you replay yesterday's ADT feed from bronze, get the same silver keys, and show that a repeated message didn't create a second row?
7. When a document-type set or FHIR profile changes, does a contract test fail before gold moves?
8. Are unmapped local codes still visible, and do medication request, dispense, administration, and statement remain separate facts?
9. Is your training set a governed snapshot with an as-of time and a purpose of use, or a file someone pulled?
10. After a feed fails over a weekend, can on-call query quarantine counts, freshness, and unusual PHI access before Monday's stand-up?

Every yes is ground you can build on, and every no is work you now know about. If you'd like a second set of eyes on your answers, that is the review our data engineering team at Light-it runs.

Why we hold ourselves to this bar

I wrote this for the engineer who gets the Slack message late at night asking why a number moved, and for the CTO who has to explain it on Monday morning.
Most of the work described here goes unseen when it goes right. Nobody celebrates a code system that made it to silver, a merge you can undo, or a replay that produces the same keys twice. You only notice when it’s missing.
That’s why we wrote the standard down. On the other end of every row are a clinician making a decision and a patient living with it. Both deserve data that means what it says. What does good data architecture in healthcare actually look like? It looks like one that wont affect the standard of care.