Medallion Architecture for Healthcare: How to Build Bronze, Silver and Gold

Updated October 2026. Medallion architecture in healthcare organizes clinical, claims and operational data into three layers: Bronze keeps what each source sent, Silver holds checked and versioned records, and Gold holds the tables people and models use. The layers are the easy part. Whether your numbers can be trusted depends on four things the pattern does not give you: patient identity, code and unit mapping, a rule for late and corrected data, and one governed definition for each metric.

This guide is for CIOs, CTOs and heads of data who have to decide what to build and who to hire. It takes a position on each of those four, shows the build step by step, and works through an example in SQL that we ran. If you already want help, see our healthcare data engineering services, including Databricks, Snowflake, data pipelines and data warehouse work.

Our position in six sentences

Medallion is a way to name quality levels, and it does not amount to a data strategy. Treat identity, reference maps and metric definitions as products with owners that sit beside the three layers. Bronze is your most sensitive store, so very few people should read it. Never overwrite a corrected result: keep every version and make each published number reproducible. Do not use FHIR as your analytics model, even if FHIR is your best feed. Build no dashboard until someone has named the decision it serves and who acts on it.

What each layer holds in a healthcare data lakehouse architecture

Each layer answers a different question: what did the source send, what do we believe, and what should the business use. Databricks describes the bronze layer as minimally validated and not meant for analysts, and advises building silver from bronze rather than writing to it straight from ingestion. Microsoft calls medallion the recommended design approach for Fabric, and recommends a separate workspace per layer for tighter governance.

Three layers move the data. The three products beside them give it meaning, and each needs a named owner.

Why build a medallion architecture in healthcare

Build it when several sources must agree and you need to explain any number months later. Healthcare data has properties that make this harder than most industries: results get corrected after they are final, claims get replaced and payments reversed, patients get merged, and code sets differ by site. A layered design gives you a place to keep the original, a place to apply each rule, and a way to replay when a rule changes.

• Reprocessing. When a mapping or a measure definition changes, rebuild from Bronze without asking the source to send years of data again.
• Explaining a number. When a quality lead questions a figure, you can walk from Gold to the original message.
• Mixed sources. FHIR, HL7 v2, X12, Epic Clarity and files from an ERP such as Oracle Fusion can land the same way and meet in Silver.
• Models that hold up. Versioned Silver lets you rebuild what was known on a past date, which prediction work needs.

When not to build it. If you run one EHR, have only reporting needs, and Epic's own Clarity and Caboodle answer your questions, a new lakehouse may add cost without adding answers. Our guide to warehouse, lake, lakehouse and mesh covers that choice. Enterprise data lake setup for healthcare organizations should start from the questions that one system cannot answer.

The dashed line matters most: what people do with the output returns as a source. Without it, you cannot tell whether anything worked.

How to build Bronze: ingest FHIR, HL7 v2, X12 and Clarity without changing them

Build Bronze as one append-only table per source that stores the payload unchanged, with arrival metadata. Databricks recommends storing most bronze fields as string, VARIANT or binary to guard against schema changes. Microsoft recommends keeping the original format whenever possible.

1. List sources and write a contract for each. Include owner, format, expected arrival, natural key, how corrections appear, and PHI class.
2. Add the same metadata everywhere. Source, file or message ID, arrival time, load batch, parser version.
3. Parse only what routing needs. Message type and resource type are fine. Leave cleanup for Silver.
4. Make loads safe to repeat. Interfaces resend and jobs retry, so duplicates in Bronze are normal. Silver removes them.
5. Lock it down before the first PHI row. Separate storage and roles, encryption, audit logs, and a signed BAA with the platform.

Be careful with partitioning. Microsoft discourages partitioning bronze in new Fabric work and suggests Liquid Clustering for silver and gold. Check your platform's current guidance, and pick clustering from real query patterns. For incremental pulls from a database, see change data capture for healthcare warehouses.

How to build Silver: clean, conform and keep every version

Build Silver by turning each raw row into typed records, sending failures to quarantine with a reason, and adding corrections as new versions. Silver is where most of the engineering and nearly all of the arguments happen.

1. Apply cleanup rules in a fixed order. Remove exact resends, parse types, standardize time zones and null markers, then validate. Keep the original value next to the cleaned one.
2. Flatten FHIR on purpose. One Silver table per resource you use is a good start. Lift extensions you analyze, such as race and ethnicity, into columns, and keep the rest as JSON.
3. Map through reviewed tables. Codes, units, statuses, facilities and payers each get a mapping table with an owner and effective dates. A CASE statement in a job is invisible to the people who should approve it.
4. Attach a person key from your identity product. Do not merge records in place.
5. Quarantine, then count. Bronze rows must equal Silver versions plus quarantined rows plus exact resends. If they do not, stop.

Dimension mapping: a variation matrix to build first

The same fact looks different in each source. Build a source-to-target mapping matrix before writing transforms, because this is the real scope of the project.

Slowly changing dimensions: decide per attribute

Use type 2 history when a past report must still reflect the past, and type 1 when an old value was simply wrong. A facility's region deserves type 2, because last quarter's East report should not move when a clinic joins North. A misspelled facility name is type 1. A patient's address is type 2 if you study access or outreach, and it is also more PHI to protect. Databricks AUTO CDC, which replaces APPLY CHANGES in Lakeflow pipelines (earlier called Delta Live Tables), handles type 1 and 2 and out-of-order arrival through a sequence column. This sketch uses the documented syntax with our table names. We have not run it on a workspace.

CREATE OR REFRESH STREAMING TABLE dim_facility;
CREATE FLOW facility_history AS AUTO CDC INTO dim_facility
FROM stream(silver.facility_changes)
KEYS (facility_key)
SEQUENCE BY source_changed_at
STORED AS SCD TYPE 2;

Identity matching: automatic only with strong evidence

Create a person key in a separate product, link source IDs to it with evidence, and send anything uncertain to a person. A wrong merge shows one patient another's data, which is worse than a missed link. The FHIR Patient resource expresses the result through link types such as replaced-by and seealso. Unmerges must be possible and must restate Gold. More on methods in patient matching beyond demographics.

Two different IDs from the same source usually mean two people, however close the demographics are.

How to build Gold: publish numbers you can replay

Build Gold from the decisions people make, as star schemas and feature tables that record what was published and when. Every Gold table starts from a question with an owner. If nobody can name the question, the table waits.

1. Fix the grain. One row per what? Patient-week, encounter, claim line. Most disputes are grain disputes.
2. Join dimensions as of the event date unless the question asks for today's structure.
3. Store a snapshot per publish with the input cut-off, code version and map version.
4. Serve other models as separate Gold products. OMOP for research and FHIR for exchange can sit next to your star schema. See OMOP vs FHIR.
5. Add event times to features, so a model can only see what was known then. See healthcare feature stores.

Why not make Silver pure FHIR? FHIR is designed to exchange records, and it does that well. Analytics needs stable grains, history and joins, which nested resources make awkward. We would use FHIR as a feed and as an output, and keep a conformed model in the middle. A team whose work is mostly FHIR exchange, with little reporting, could reasonably decide otherwise.

Worked example: a number that changes three times

This example is illustrative: all patients, facilities, values and dates are synthetic, and the 180 mg/dL threshold and matching weights are made up for teaching. We ran it on SQLite 3.51 on 31 October 2026. The blocks below run in order as one script, and the outputs are copied from that run. SQLite has no MERGE or Delta features, so treat this as logic to port rather than production code. The question: how many people have a latest glucose above 180 mg/dL?

Bronze. Twelve rows from two feeds, including a resend, a correction, a late file in other units, a withdrawn result, and three bad rows.

-- Bronze: append-only. Every row keeps what the source sent, as text, plus when it arrived.
CREATE TABLE bronze_lab (
  row_id INTEGER PRIMARY KEY, source TEXT, msg_id TEXT, order_id TEXT, local_patient TEXT,
  facility_raw TEXT, code_raw TEXT, value_raw TEXT, unit_raw TEXT, status_raw TEXT,
  observed_at TEXT, received_at TEXT);
INSERT INTO bronze_lab VALUES
(1 ,'hl7_lab'    ,'M1','ORD1' ,'H100','MAIN-LAB'         ,'GLU'    ,'250'     ,'mg/dL' ,'F'    ,'2026-03-02 08:00','2026-03-02 08:05'),
(2 ,'hl7_lab'    ,'M1','ORD1' ,'H100','MAIN-LAB'         ,'GLU'    ,'250'     ,'mg/dL' ,'F'    ,'2026-03-02 08:00','2026-03-02 08:06'), -- resend
(3 ,'hl7_lab'    ,'M2','ORD1' ,'H100','MAIN-LAB'         ,'GLU'    ,'125'     ,'mg/dL' ,'C'    ,'2026-03-02 08:00','2026-03-03 16:00'), -- correction
(4 ,'fhir_clinic','B1','OBS9' ,'C-77','EASTSIDE'         ,'14749-6','11.1'    ,'mmol/L','final','2026-03-01 10:00','2026-03-08 01:00'), -- late, other unit
(5 ,'hl7_lab'    ,'M3','ORD3' ,'H200','Main Hospital Lab','GLU'    ,'190'     ,'mg/dL' ,'F'    ,'2026-03-04 07:00','2026-03-04 07:10'),
(6 ,'hl7_lab'    ,'M4','ORD4' ,'H200','MAIN-LAB'         ,'GLU'    ,'see note','mg/dL' ,'F'    ,'2026-03-04 09:00','2026-03-04 09:05'), -- not a number
(7 ,'fhir_clinic','B2','OBS10','C-80','Northgate'        ,'2345-7' ,'240'     ,'mg/dL' ,'final','2026-03-06 10:00','2026-03-08 01:00'),
(8 ,'fhir_clinic','B3','OBS11','C-81','Northgate'        ,'2345-7' ,'130'     ,''      ,'final','2026-03-06 11:00','2026-03-08 01:00'), -- no unit
(9 ,'hl7_lab'    ,'M5','ORD5' ,'H400','MAIN-LAB'         ,'GLUX'   ,'88'      ,'mg/dL' ,'F'    ,'2026-03-05 07:00','2026-03-05 07:05'), -- unknown code
(10,'hl7_lab'    ,'M6','ORD6' ,'H500','MAIN-LAB'         ,'GLU'    ,'300'     ,'mg/dL' ,'F'    ,'2026-03-05 08:00','2026-03-05 08:05'),
(11,'hl7_lab'    ,'M7','ORD6' ,'H500','MAIN-LAB'         ,'GLU'    ,'300'     ,'mg/dL' ,'W'    ,'2026-03-05 08:00','2026-03-06 09:00'), -- posted to wrong patient
(12,'hl7_lab'    ,'M8','ORD7' ,'H600','MAIN-LAB'         ,'GLU'    ,'110'     ,'mg/dL' ,'F'    ,'2026-03-09 11:00','2026-03-09 11:30');

Reference data. Note the facility that changes name and region on 5 March.

-- Reference data the business owns. Each mapping is a reviewed row, never a CASE buried in code.
CREATE TABLE map_code (source TEXT, code_raw TEXT, loinc TEXT);
INSERT INTO map_code VALUES ('hl7_lab','GLU','2345-7'),('fhir_clinic','2345-7','2345-7'),('fhir_clinic','14749-6','14749-6');
CREATE TABLE map_unit (loinc TEXT, unit_raw TEXT, to_mg_dl REAL);
INSERT INTO map_unit VALUES ('2345-7','mg/dL',1.0),('14749-6','mmol/L',18.016);
CREATE TABLE map_status (status_raw TEXT, status TEXT);
INSERT INTO map_status VALUES ('F','final'),('C','corrected'),('W','withdrawn'),('final','final'),('corrected','corrected'),('entered-in-error','withdrawn');
CREATE TABLE map_facility (facility_raw TEXT, facility_key TEXT);
INSERT INTO map_facility VALUES ('MAIN-LAB','F1'),('Main Hospital Lab','F1'),('EASTSIDE','F2'),('Northgate','F2');
-- Slowly changing dimension, type 2: F2 was renamed and moved region on 5 March.
CREATE TABLE dim_facility (facility_key TEXT, name TEXT, region TEXT, valid_from TEXT, valid_to TEXT);
INSERT INTO dim_facility VALUES
('F1','Main Hospital'  ,'Central','2020-01-01','9999-12-31'),
('F2','Eastside Clinic','East'   ,'2020-01-01','2026-03-05 00:00'),
('F2','Northgate Clinic','North' ,'2026-03-05 00:00','9999-12-31');

Identity. This query scores pairs that share a birth date.

-- Candidate matches. Weights and cut-offs are illustrative and need tuning on your own reviewed pairs.
CREATE TABLE patient_record (source TEXT, local_patient TEXT, first TEXT, last TEXT, dob TEXT, zip TEXT);
INSERT INTO patient_record VALUES
('hl7_lab'    ,'H100','Ana' ,'Sample','1961-04-12','60601'),
('fhir_clinic','C-77','Anna','Sample','1961-04-12','60614'), -- same person, moved
('hl7_lab'    ,'H300','Ann' ,'Sample','1961-04-12','60601'), -- her twin sister
('fhir_clinic','C-80','Raj' ,'Testa' ,'1975-09-30','60610');
SELECT a.local_patient AS a, b.local_patient AS b,
  (a.dob = b.dob) * 40 + (a.last = b.last) * 30
  + (substr(a.first,1,2) = substr(b.first,1,2)) * 10 + (a.zip = b.zip) * 20 AS score,
  CASE WHEN a.source = b.source THEN 'review: same source, two IDs'
       WHEN (a.dob = b.dob) * 40 + (a.last = b.last) * 30
          + (substr(a.first,1,2) = substr(b.first,1,2)) * 10 + (a.zip = b.zip) * 20 >= 90 THEN 'auto-link'
       WHEN (a.dob = b.dob) * 40 + (a.last = b.last) * 30
          + (substr(a.first,1,2) = substr(b.first,1,2)) * 10 + (a.zip = b.zip) * 20 >= 60 THEN 'review'
       ELSE 'no link' END AS decision
FROM patient_record a JOIN patient_record b ON a.local_patient < b.local_patient
WHERE a.dob = b.dob;

a | b | score | decision
H100 | H300 | 100 | review: same source, two IDs
C-77 | H100 | 80 | review
C-77 | H300 | 80 | review

H100 and H300 score a perfect 100 and are different people: they are twins at one address. A score alone would have merged them. The same-source rule sent them to review. H100 and C-77 are one person who moved, with a lower score of 80. In our synthetic story, a reviewer links those two and keeps H300 apart, and this table records the result.

-- A person reviewed the pairs: H100 and C-77 are one person. H300 is not.
CREATE TABLE person_xref (local_patient TEXT, person_key TEXT, how TEXT);
INSERT INTO person_xref VALUES
('H100','P1','human review'),('C-77','P1','human review'),('H300','P3','human review'),
('H200','P2','single record'),('C-80','P4','single record'),('H500','P5','single record'),('H600','P6','single record');

Silver. Every valid version stays, and failures carry a reason.

-- Silver keeps every valid version. Rows that fail a rule go to quarantine with the reason.
CREATE VIEW checked AS
SELECT b.*, c.loinc, u.to_mg_dl, s.status,
  CASE WHEN c.loinc IS NULL THEN 'unmapped code'
       WHEN b.unit_raw = '' THEN 'missing unit'
       WHEN u.to_mg_dl IS NULL THEN 'unmapped unit'
       WHEN b.value_raw GLOB '*[^0-9.]*' OR b.value_raw = '' THEN 'not numeric'
       WHEN s.status IS NULL THEN 'unmapped status' END AS reject_reason
FROM bronze_lab b
LEFT JOIN map_code c ON c.source = b.source AND c.code_raw = b.code_raw
LEFT JOIN map_unit u ON u.loinc = c.loinc AND u.unit_raw = b.unit_raw
LEFT JOIN map_status s ON s.status_raw = b.status_raw;

CREATE TABLE quarantine AS SELECT row_id, source, msg_id, reject_reason FROM checked WHERE reject_reason IS NOT NULL;

CREATE TABLE silver_lab AS
SELECT min(row_id) AS row_id, source, msg_id, order_id, x.person_key, f.facility_key, 'glucose' AS test,
       round(CAST(value_raw AS REAL) * to_mg_dl, 1) AS mg_dl, status, observed_at, min(received_at) AS received_at
FROM checked c
JOIN person_xref x ON x.local_patient = c.local_patient
JOIN map_facility f ON f.facility_raw = c.facility_raw
WHERE reject_reason IS NULL
GROUP BY source, msg_id;  -- a resent message counts once

Gold. The same question, asked at three moments.

-- Gold: the number as anyone would have seen it at a chosen moment.
-- Rule 1: per order, the newest version received by then wins. Rule 2: per person, the latest observation wins.
CREATE TABLE publish_run (as_of TEXT);
INSERT INTO publish_run VALUES ('2026-03-02 12:00'),('2026-03-04 12:00'),('2026-03-09 12:00');

CREATE VIEW latest_by_person AS
WITH v AS (
  SELECT r.as_of, s.*, row_number() OVER (PARTITION BY r.as_of, s.source, s.order_id ORDER BY s.received_at DESC) AS rn
  FROM publish_run r JOIN silver_lab s ON s.received_at <= r.as_of),
o AS (
  SELECT *, row_number() OVER (PARTITION BY as_of, person_key ORDER BY observed_at DESC) AS pn
  FROM v WHERE rn = 1 AND status <> 'withdrawn')
SELECT * FROM o WHERE pn = 1;

SELECT as_of, count(*) AS people_with_a_result, sum(mg_dl > 180) AS over_180_mg_dl,
       group_concat(CASE WHEN mg_dl > 180 THEN person_key END) AS who
FROM latest_by_person GROUP BY as_of;

as_of | people_with_a_result | over_180_mg_dl | who
2026-03-02 12:00 | 1 | 1 | P1
2026-03-04 12:00 | 2 | 1 | P2
2026-03-09 12:00 | 4 | 2 | P2,P4

On 2 March the answer is P1, whose result reads 250. On 3 March the lab corrects it to 125, so by 4 March P1 is out and P2 is in. On 8 March the clinic's weekly file brings a 1 March result of 11.1 mmol/L for the same person as P1, about 200 mg/dL. It does not put P1 back, because the corrected 2 March result is newer. Each answer was right for what was known at the time, and all three can be reproduced.

What if nobody had linked the patient?

-- The same number if nobody had linked H100 and C-77: the two sources count as two people.
SELECT count(*) AS over_180_without_identity_link FROM (
  SELECT mg_dl, row_number() OVER (PARTITION BY source, person_key ORDER BY observed_at DESC) AS pn
  FROM (
    SELECT s.*, row_number() OVER (PARTITION BY source, order_id ORDER BY received_at DESC) AS rn
    FROM silver_lab s WHERE received_at <= '2026-03-09 12:00')
  WHERE rn = 1 AND status <> 'withdrawn')
WHERE pn = 1 AND mg_dl > 180;

over_180_without_identity_link
3

The count becomes 3 instead of 2. The extra person is P1, whose older clinic result now looks like a current one.

Slowly changing dimension. The late result belongs to a clinic that has since moved region.

-- Which region gets the Clinic result from 1 March? Both answers are right; they answer different questions.
SELECT s.order_id, s.observed_at,
  (SELECT d.region FROM dim_facility d WHERE d.facility_key = s.facility_key
     AND s.observed_at >= d.valid_from AND s.observed_at < d.valid_to) AS region_when_observed,
  (SELECT d.region FROM dim_facility d WHERE d.facility_key = s.facility_key AND d.valid_to = '9999-12-31') AS region_today
FROM silver_lab s WHERE s.order_id = 'OBS9';

order_id | observed_at | region_when_observed | region_today
OBS9 | 2026-03-01 10:00 | East | North

X12 corrections.

-- X12 corrections. Summing every row is wrong; each claim needs its latest 837 and its net 835.
CREATE TABLE bronze_x12 (seq INTEGER, tx TEXT, claim TEXT, code TEXT, amount REAL);
INSERT INTO bronze_x12 VALUES
(1,'837','CLM1','1' ,1200),  -- CLM05-3 = 1, original
(2,'837','CLM1','7' , 900),  -- CLM05-3 = 7, replacement
(3,'835','CLM1','1' , 700),  -- CLP02 = 1, paid
(4,'835','CLM1','22',-700),  -- CLP02 = 22, reversal
(5,'835','CLM1','1' , 650);  -- corrected payment
SELECT claim,
  sum(CASE WHEN tx='837' THEN amount END) AS naive_charge,
  (SELECT amount FROM bronze_x12 x WHERE x.claim=b.claim AND x.tx='837' ORDER BY seq DESC LIMIT 1) AS current_charge,
  sum(CASE WHEN tx='835' AND amount>0 THEN amount END) AS paid_ignoring_reversal,
  sum(CASE WHEN tx='835' THEN amount END) AS net_paid
FROM bronze_x12 b GROUP BY claim;

claim | naive_charge | current_charge | paid_ignoring_reversal | net_paid
CLM1 | 2100.0 | 900.0 | 1350.0 | 650.0

Adding every row reports 2,100 in charges against a true 900, and 1,350 paid against a true 650.

Checks before release.

-- Checks that run before Gold publishes. A failure holds the release and tells the owner.
CREATE TABLE source_contract (source TEXT, owner TEXT, max_hours REAL, last_received TEXT);
INSERT INTO source_contract VALUES
('hl7_lab','Interfaces team',2,(SELECT max(received_at) FROM bronze_lab WHERE source='hl7_lab')),
('fhir_clinic','Clinic IT',168,(SELECT max(received_at) FROM bronze_lab WHERE source='fhir_clinic')),
('x12_835','Revenue cycle',168,'2026-02-20 06:00');

SELECT source, owner, last_received,
  round((julianday('2026-03-09 12:00') - julianday(last_received)) * 24, 1) AS hours_old, max_hours,
  CASE WHEN (julianday('2026-03-09 12:00') - julianday(last_received)) * 24 > max_hours THEN 'STALE' ELSE 'ok' END AS status
FROM source_contract;

SELECT (SELECT count(*) FROM bronze_lab) AS bronze_rows,
       (SELECT count(*) FROM silver_lab) AS silver_versions,
       (SELECT count(*) FROM quarantine) AS quarantined,
       (SELECT count(*) FROM bronze_lab) - (SELECT count(*) FROM silver_lab) - (SELECT count(*) FROM quarantine) AS exact_resends,
       (SELECT group_concat(reject_reason, '; ') FROM quarantine) AS reasons;

source | owner | last_received | hours_old | max_hours | status
hl7_lab | Interfaces team | 2026-03-09 11:30 | 0.5 | 2.0 | ok
fhir_clinic | Clinic IT | 2026-03-08 01:00 | 35.0 | 168.0 | ok
x12_835 | Revenue cycle | 2026-02-20 06:00 | 414.0 | 168.0 | STALE

bronze_rows | silver_versions | quarantined | exact_resends | reasons
12 | 8 | 3 | 1 | not numeric; missing unit; unmapped code

The row counts reconcile: 12 = 8 + 3 + 1. The remittance feed is 414 hours old against a 168-hour limit, so any revenue figure should hold or show its "data through" date. Clinical figures can still release.

The challenges every team sees coming

The visible challenges are sources on different clocks, late data, messy values and sensitive data. Each has a workable pattern.

Forcing all feeds onto one schedule hides staleness. Showing each source's freshness exposes it.

The real issues, which rarely appear on the project plan

The real issues are about ownership and meaning, and tools do not fix them. These are the ones we would worry about first. They are our judgement and are not survey findings.

• Nobody owns the definition. Finance, quality and operations each have their own "readmission" or "denial rate". All three can be defensible. The platform makes the disagreement more visible, and only an owner settles it.
• The EHR is treated as one source. Two hospitals on one vendor can differ in build, codes and workflow. Scope by site and by feed.
• Silver quietly becomes the business logic. Rules land in jobs nobody outside engineering reads. Move what the business must approve into reviewed tables.
• Quality checks test shape, not truth. A check that a value is numeric does not show that it is right. Reconcile to something outside the pipeline, such as a lab's own monthly count or a finance ledger.
• Unmapped rates look fine by row and bad by meaning. A small share of rows can hide the one code a measure depends on. Report unmapped rates per measure.
• Identity is left to the end. Every count of people depends on it, so retrofitting it restates everything.
• Dashboards outlive their purpose. Nobody retires them, so nobody knows which ones matter.

How to solve them: contracts, lineage, checks and restatement

Solve them with four habits: a contract per source, a test at each boundary, lineage you can query, and a written rule for restating numbers. Together they make each failure visible and give it an owner.

Each row ends in exactly one place, so counts can always reconcile.

Data contracts

A data contract is a written, testable agreement between a source owner and the platform. Cover owner and backup; format and version; maximum age; keys; required fields; allowed codes and units; how corrections and deletes appear; PHI class; and notice before changes. Run it as code at the Bronze to Silver boundary. Data quality at scale covers what integration does to quality.

Quality checks, and what each one actually catches

On Databricks, expectations can warn, drop or fail a run; the docs treat quarantine as an advanced pattern you build yourself. We would not use drop for clinical rows, because a dropped row leaves no reason behind. On Snowflake, data metric functions measure attributes such as null counts. In Fabric, materialized lake views support built-in quality rules.

Lineage

Lineage should answer two questions quickly: where did this number come from, and what breaks if this mapping changes? Unity Catalog captures table and column lineage, with stated gaps. Column lineage is best effort, UDF lineage is limited, renames do not carry lineage over, and its system tables keep a rolling year. So platform lineage shows that a table fed a table. It does not show that a rule was right. Store batch, code version and map version on each published row as well, and export lineage if your audit horizon is longer than a year.

PHI by layer

Control PHI by layer, by role and by purpose. HIPAA's minimum necessary standard asks for reasonable efforts to limit use, and its audit controls standard asks you to record and examine activity in systems holding electronic PHI. Have your privacy officer and counsel confirm your specific policy.

Four details are easy to get wrong:

• "De-identified" has a legal meaning. Safe Harbor removes 18 identifier types, including all date elements except year, and allows ZIP code digits only under its 20,000-person rule. Hashing an MRN does not meet that by itself, and a Gold table with admission dates does not qualify under Safe Harbor. Otherwise you need Expert Determination.
• Six years is about documentation. HIPAA's six-year rule covers required policies and records of actions. Medical record retention comes mainly from state and program rules. Set Bronze retention with counsel rather than assuming a number. An earlier version of this guide stated seven years; we have removed that.
• Time travel is not retention. Delta history depends on files that VACUUM removes, with a seven-day default threshold. Conversely, a DELETE alone does not remove PHI from old files until VACUUM runs.
• Platform settings are part of the BAA story. Databricks ties HIPAA controls to its compliance security profile. Snowflake's Dynamic Data Masking applies at query time and requires Enterprise Edition or higher. Read current terms for each service before PHI lands.

Semantic models: one definition for each metric

A semantic model is the layer where a metric gets one name, one formula, one grain and one owner, so that every dashboard and AI tool gets the same answer. Without it, each BI report re-implements the logic, and the numbers slowly diverge. This is the layer we would fund before adding more dashboards.

Keep metrics apart from official measure programs. A HEDIS or CMS measure has its own specification and rules, which you should license or follow exactly. The illustrative threshold above is not a quality measure.

A Databricks metric view looks like this. It follows the documented YAML shape with our names, and we have not run it on a workspace.

version: 1.1
comment: 'Owner: quality lead. v3. Illustrative threshold.'
source: gold.latest_glucose_by_person
fields:
  - name: Region
    expr: region_when_observed
measures:
  - name: People over threshold
    expr: COUNT(DISTINCT person_key) FILTER (WHERE mg_dl > 180)

Choose one place where definitions live and have BI read from it. If you run two platforms, pick the one that owns each metric and test that the other returns the same number. Semantic definitions also give AI assistants something safer to query than raw tables, though they do not make a generated answer correct.

Dashboards people can act on

A useful dashboard answers one decision for one role and leads to a queue of named work. If the number moves and nobody would do anything differently, drop the dashboard.

• State the decision in the title. "Which patients need follow-up today" beats "Lab Overview".
• Link each number to its worklist, with role-limited access to patient detail.
• Show freshness and restatements. "Data through" per source, and a mark when a past value changed.
• Set a threshold and an owner. What level triggers action, and who acts by when.
• Record the action. It becomes data for the next step.
• Review and retire on a schedule. Ask each owner whether it still drives a decision.

The prediction and forecasting loop

Predictions belong in the same system as everything else: features from Gold, scores stored as data, and outcomes returned to Bronze so that error can be measured. The point-in-time rule is easy to break.

-- Training data must use what was known then. Evaluation uses what turned out to be true.
SELECT '2026-03-02 12:00' AS prediction_time,
  (SELECT mg_dl FROM silver_lab WHERE order_id='ORD1' AND received_at <= '2026-03-02 12:00' ORDER BY received_at DESC LIMIT 1) AS value_known_then,
  (SELECT mg_dl FROM silver_lab WHERE order_id='ORD1' ORDER BY received_at DESC LIMIT 1) AS value_known_now;

prediction_time | value_known_then | value_known_now
2026-03-02 12:00 | 250.0 | 125.0

A model scoring at noon on 2 March saw 250. Training it on 125 teaches it from information it could not have had. Databricks' point-in-time feature joins exist for this kind of leakage. Versioned Silver makes it possible, which is why overwriting corrections hurts more than reporting. Use the value known then as the input, and the corrected value when judging the outcome.

Store each score with its model version, input cut-off and prediction time. Otherwise you cannot audit it later.

• Start with forecasts that have a clear owner. Staffing against expected census or planning for denied claims volume are examples. Compare against a simple baseline such as last year's same week. If the model cannot beat it, keep the baseline.
• Measure error by group, including facility and patient population, and watch for drift.
• Account for your own actions. If outreach prevents an event, the outcome data will make the model look wrong. Record who was acted on, and keep a fair comparison if the risk is material.
• Keep people in clinical decisions, and check with counsel and regulatory staff when a model informs care.

For the path from data to a deployed model, see healthcare ML pipelines with an audit trail.

Databricks, Snowflake or Fabric for a healthcare medallion

All three can carry the pattern, so choose on your team's skills, existing contracts and workload. This is our judgement, and none of them solves identity or definitions for you.

A dated warning on Fabric. Microsoft says that from 1 October 2026 only existing customers can deploy the managed healthcare data solutions, new customers should use a downloadable source package, and managed support ends 31 December 2027. If you are comparing on that accelerator, plan to own that code. For a FHIR to Databricks ETL pipeline in detail, see our healthcare data lakehouse guide.

What a decision-maker should ask before hiring anyone

Judge a vendor by how they handle corrections, identity and ownership, because any competent team can draw three boxes. This applies to us as well.

In our judgement, ownership gaps cause more trouble than tool choice. Put a person's name in each box.

What we can show you. 350+ Mirth Connect interfaces, an open-source X12 clearinghouse simulator, and 20 public healthcare repositories you can read. This guide does not cite a data platform project of ours, so hold us to the questions above.

Healthcare Data Platform Assessment: fixed $3,500

The fastest way to find out what this would take for you is our Healthcare Data Platform Assessment, a fixed $3,500 review that takes 2 to 3 weeks and is credited against the build. You receive:

• A source inventory
• A target architecture
• A data model outline
• A risk list
• A costed build plan

What we need from you: a named contact, your source and interface list, the reports that matter most, and schema or sample structure without PHI. We do not need production PHI for this review.

Not included: building pipelines, handling production PHI, platform licenses and cloud charges, EHR or interface vendor fees, legal or compliance sign-off, and running an official quality measure. You can take the plan to any team.

Request the Healthcare Data Platform Assessment or read more on our healthcare data engineering page.