Databricks for Healthcare
Databricks pays off when your data is large and messy or AI is central to your plans. A poorly run workspace brings unclear costs and loose access to patient data. We give you one governed platform for reporting and AI, with an owner for each cost and access you approve. A fixed-price assessment shows the plan, the cost and whether Databricks is the right choice.
Book the Healthcare Data Platform AssessmentFor data leaders building or improving a Databricks environment.
The platform is bought, but costs, access to patient data and trust in the numbers are unclear.
One governed platform for reporting and AI, with an owner for each number and cost.
Patient data stays in your account, with the required agreements and access limits in place first.
$3,500 fixed for the assessment. 2 to 3 weeks, credited against the build.
Book the assessment. If Databricks is the wrong fit, it will say so.
The business case
The software is rarely the problem. How it is set up and who controls it decides whether it pays back.
Healthcare data is large, nested and full of corrections. Databricks can handle it, but only when the setup is planned for patient data and someone owns the definitions. Without that, you pay for compute while reports still disagree and AI projects cannot get data they may safely use.
Our view
Databricks earns its place when you have large nested FHIR and HL7 data, streaming, or machine learning on the same data your analysts use. If your work is mostly SQL reporting and partner sharing, compare it honestly with Snowflake. Whichever you pick, set up the compliance profile and Unity Catalog before the first PHI load.
Databricks states that you must have a BAA before processing PHI, and that confirming each workspace has the compliance security profile enabled is your responsibility. We treat that as a first-day check.
Databricks tells customers not to use preview features with PHI unless they are on its supported list. New features look attractive in a demo, so we check the list before designing around one.
Databricks documents that table-level row filters and column masks do not support time travel, clones, or being applied to a view. That affects audit and recovery plans, so the access design has to come early.
Lakeflow expectations can warn, drop or fail. Quarantine is a pattern you build, and healthcare needs it, because silently dropping a lab result is rarely acceptable.
When we would tell you not to choose Databricks: your team is SQL-first, the data is mostly structured and your main need is reporting or sharing with partners. Snowflake or your existing warehouse may be the simpler choice.
For decision makers
These are the questions that decide whether a data engineering vendor is safe to hire. Here are our answers, and how to test them.
| Risk | How we control it |
|---|---|
| PHI lands in an unprepared workspace | A written readiness check before the first load, for each workspace |
| A preview feature ends up in a PHI path | Check the design against the Databricks supported list and record the result |
| Records dropped without notice | A quarantine pattern with counts that tie to the event log |
| Compute spend climbs | Compute policies, a workload owner and refresh rates tied to decisions |
| Access rules break on some compute | Test filters and masks on the runtimes and access modes you will really use |
The first step
Fixed price: $3,500. 2 to 3 weeks, credited against the build.
A fixed-price first step for one decision you need the data to support. You finish with enough evidence to approve, change or stop the build, and the outputs are yours to use with any vendor.
Book the Healthcare Data Platform AssessmentThe assessment is complete when all five outputs are delivered and walked through with your owners. Sending the form starts a conversation, and you pay nothing until you agree the assessment.
How the work runs
The order matters here: governance and environments come before pipelines, because PHI should never land in a workspace that has not been checked.
Confirm the BAA, compliance security profile, customer-managed keys or notebook-results setting, and your cloud BAA.
You review: A signed-off readiness record
Set catalogs per environment, schemas per domain, service principals and groups.
You review: A catalog plan with access tests
Set up CI/CD with Databricks Asset Bundles so that environments can be rebuilt.
You review: A repeatable deployment
Ingest with Auto Loader, then build silver and gold in Lakeflow with expectations and AUTO CDC.
You review: Delta tables with quality metrics
Define measures and dimensions in Unity Catalog metric views, then connect dashboards.
You review: Governed metrics
Monitor with the pipeline event log and data profiling, review compute costs and rehearse recovery.
You review: A runbook and a cost review
Related delivery
Public-source life-sciences data moves through collection, name reconciliation and approval before reporting and source-cited AI answers.
Delivery context: a public-data intelligence platform built with Frappe and Power BI, with AI assistants. The case shows source integration, reconciliation and review workflows.
Read the implementation case study →
Cost drivers
Engineering effort and platform consumption move together, so we review both.
FHIR NDJSON, HL7 v2, X12 and database CDC each need their own parsing and tests.
Continuous streaming jobs cost more to build and run than scheduled batch or triggered pipelines.
Migrating from the legacy Hive metastore to Unity Catalog or untangling notebooks adds work before new features.
Backfill depth and daily change volume shape both engineering time and compute.
Row filters, column masks, multiple environments and lineage evidence add setup and review work.
Feature tables, evaluation sets and MLflow tracking are scoped separately from the core pipelines.
For your engineers
What your technical team will want to check: ingestion, layers, governance on Databricks and samples of the work. Platform statements were read on the Databricks documentation in October 2026.
The short answer
We implement Databricks lakehouses for healthcare data: FHIR Bulk Data NDJSON and HL7 v2 feeds land in bronze Delta tables through Auto Loader, get validated and flattened in silver with Lakeflow pipeline expectations, and publish as gold patient, encounter, claim and cohort tables. Unity Catalog governs access to PHI with row filters, column masks and lineage. PHI workloads run in workspaces with the compliance security profile enabled and a Databricks BAA in place.
What we build
Named Databricks features, applied to healthcare sources and their update behavior.
Auto Loader ingests Bulk Data $export files per resource type into bronze Delta tables with file, export and resource version metadata. Silver tables flatten nested FHIR JSON (Patient, Encounter, Observation, Condition, MedicationRequest, Claim) with explicit handling of extensions and contained resources.
FHIR-to-Databricks pipeline designBronze keeps the source as received, silver applies schema, identifier and terminology checks, and gold publishes consumer models. Lakeflow pipeline expectations (formerly Delta Live Tables) can warn, drop or fail a record. Quarantine is a pattern we add, so failed records stay visible with a reason.
Medallion architecture for healthcareCatalogs per environment, schemas per domain, service principals for jobs, row filters and column masks for PHI, and table and column lineage. Access is tested with the same roles that will consume the data.
Governance, lineage and PHI accessChange data feed and AUTO CDC, which replaces APPLY CHANGES (SCD type 1 or 2), for database CDC streams, corrected lab results, claim adjustments and reversals, and a replay test checks that reprocessing does not double count.
CDC for healthcare data warehousesHL7 v2 messages from an interface engine or Kafka topic parsed to segment-level tables, and X12 837 and 835 files parsed to claim, service line and adjustment tables with CARC and RARC codes.
Streaming healthcare data with Kafka and FHIRVersioned feature tables and evaluation sets tracked with MLflow, plus de-identified gold tables prepared under HIPAA Safe Harbor or for an Expert Determination.
Feature stores for clinical dataSources handled
Each source maps to a specific way in. The healthcare detail lies in what you keep alongside the payload.
| Source | Databricks path | What we keep or decide |
|---|---|---|
| FHIR Bulk Data NDJSON | Auto Loader from cloud storage into bronze | Export ID, file name and resource version. Bronze fields stay as string or VARIANT to absorb schema changes |
| HL7 v2 via an interface engine | Kafka or files, then Structured Streaming | The raw message and control ID, then parse to segment tables in silver |
| X12 837 and 835 | Files through Auto Loader, parsed with Spark | Claim, line and adjustment tables with CARC and RARC codes |
| EHR and billing databases | CDC feed or snapshots into AUTO CDC | A sequence column, so that late changes cannot overwrite newer ones |
| Imaging and clinical notes | Unity Catalog volumes | Metadata and access rules, with extraction scoped as separate work |
Integration boundaries
What we deliver, and what stays with your platform, security and clinical owners.
You own the Databricks account, cloud subscription and BAA. We configure workspaces, Unity Catalog and deployment within them.
EHR and claims sources grant access; we build the connectors, loads and reconciliation.
Your identity team owns SSO and groups. We map them to catalogs, schemas and service principals and test the result.
Clinical owners approve definitions such as cohorts and measures. We encode them as tested transformations.
Training, evaluation and serving are a separate stage built on the published gold tables.
Compare your options
Both run healthcare workloads well. The deciding factors are the workload shape and the team that will operate it.
| Criterion | Databricks | Snowflake |
|---|---|---|
| Nested FHIR JSON at volume | Spark and Delta handle deep nesting and large NDJSON loads natively | VARIANT and LATERAL FLATTEN work well; model the flattening deliberately |
| Streaming and CDC | Structured Streaming, Auto Loader, change data feed and AUTO CDC | Snowpipe Streaming, streams and tasks, dynamic tables |
| ML and AI | MLflow, feature tables and model serving in the same workspace | Snowpark and Cortex; many teams pair it with an external ML stack |
| Governance | Unity Catalog lineage, row filters and column masks | Masking and row access policies, object tagging, access history |
| PHI prerequisites | Compliance security profile on the workspace and a Databricks BAA | Business Critical edition or higher and a Snowflake BAA |
| Team fit | Engineers comfortable with Python, Spark and notebooks | SQL-first analysts and analytics engineers using dbt |
Many healthcare organizations run both. We design the boundary so each dataset has one owner and one definition.
Medallion layers
Databricks calls the medallion pattern a recommended practice, not a requirement. We follow its guidance, then add healthcare rules.
| Layer | Databricks guidance | Healthcare rule we add |
|---|---|---|
| Bronze | Raw, minimally validated data with full history, feeding silver and not analysts | Restrict to pipeline service principals, because it holds unmasked PHI |
| Silver | Clean, deduplicate and type the data, keeping a non-aggregated version of each record. Read from bronze instead of writing directly from ingestion | Choose how to represent nested FHIR (VARIANT, structs or normalized tables) for each resource, and quarantine failures with a reason |
| Gold | Dimensional models with pre-computed aggregates such as materialized views, by business domain | Apply row filters and column masks, and release only after reconciliation |
Dimension mapping
AUTO CDC computes SCD type 1 or type 2 tables from a change feed, or from snapshots when the source has no CDC. Each dimension needs a deliberate choice between them.
| Dimension | History choice | Why | Access treatment |
|---|---|---|---|
| Patient | SCD type 2 on demographics, with a separate merge history | Reports must explain past matches | Column masks on identifiers |
| Provider | SCD type 2 on role and group | Attribution changes over time | Standard read |
| Payer and plan | SCD type 2 with effective dates | Contracts change | Standard read |
| Location | SCD type 2 on hierarchy | Departments get reorganized | Row filters where a facility limit applies |
| Code crosswalks | SCD type 1 with a version column | Each release replaces the last, and the version stays traceable | Standard read |
Semantic models
Unity Catalog metric views let you define a measure once and group it by any available field. The engine generates the query, and SQL editors, dashboards, Genie and BI tools such as Power BI and Tableau can read it.
| Example metric | Measure in the metric view | Dimensions | Decision for its owner |
|---|---|---|---|
| Initial denial rate | Denied first-pass claims divided by adjudicated first-pass claims | Payer, reason group, facility, month | Claim or line grain, and which reason codes count |
| Average length of stay | Patient days divided by discharges | Service line, facility, discharge month | How to treat observation stays and transfers |
| Result turnaround | Median time from collection to final result | Test group, lab, hour of day | Which timestamp counts as collection |
Refresh frequency
Databricks notes the tradeoff directly: continuous ingestion costs more with lower latency, and batch costs less with higher latency. Choose for each table.
| Need | Option | Check first |
|---|---|---|
| Files arrive through the day | Auto Loader on a triggered schedule | How much delay the consumer accepts |
| ADT-driven workflow | Continuous Lakeflow pipeline | Who acts on it, and the cost of always-on compute |
| Database corrections | AUTO CDC with a sequence column | That the sequence value is reliable |
| Heavy dashboard queries | Materialized metric views on a schedule | Release status for PHI use, plus a schedule, as changing one does not itself trigger a refresh |
Cleanup problems and fixes
These are issues we look for when reviewing an existing workspace.
| Problem | Fix | How you can check it |
|---|---|---|
| Ingestion writes straight into silver | Add a bronze layer so that data can be replayed | Rebuild silver from bronze in test |
| Expectations drop records silently | Route failures to a quarantine table with a reason | Dropped counts in the event log match quarantine counts |
| Late CDC events overwrite newer ones | AUTO CDC with a reliable sequence column | Send updates out of order and check the end state |
| A workspace is missing the compliance profile | Enable it and record the check for each workspace that holds PHI | Account-level review of every workspace |
| Audit plan relies on time travel over filtered tables | Keep history in bronze or dedicated audit tables, since table-level filters do not support time travel | Rehearse a point-in-time question |
| PHI typed into job names or comments | Naming rules and review, as Databricks makes this your responsibility | Scan names and comments in customer-defined fields |
Dashboards people act on
AI/BI dashboards and Genie can read the same metric views, so a chart and a typed question give the same definition. These are scoping examples.
| Who opens it | The question | The action it triggers | Databricks piece |
|---|---|---|---|
| Revenue cycle lead | Which denial reasons grew this month? | Assign the rework queue | Metric view with an alert |
| Quality lead | Which units are drifting on a measure? | Start a review | Dashboard with row filters by facility |
| Data scientist | Has an input feature shifted? | Retrain or pause the model | Data profiling drift tables |
| Data platform owner | Which jobs cost most for what they serve? | Change the schedule or compute | Dashboard over usage data |
Forecasting loop
Start with a SQL forecast and a baseline, and move to a custom model only if the numbers justify it.
Use ai_forecast for a quick start. It returns a forecast with upper and lower bounds. Databricks marks version 1 as Public Preview and version 2 as Beta, so confirm the status before any PHI use.
Train a custom model with MLflow tracking when the simple forecast cannot meet the need.
Write each forecast to a Delta table with its run date, so that later comparison is honest.
Join forecasts to actuals and to a simple baseline, and show the error on the dashboard.
Use data profiling, which reports drift against a baseline or over time, to prompt a retrain or a retirement.
A model that influences clinical care needs clinical evaluation and its own scope.
Deliverables
All of it lives in your repository and your Databricks account.
BAA, compliance profile and encryption checks for each workspace.
Pipelines, jobs and environments defined as code.
Catalogs, groups, row filters, column masks and test results.
Signed-off measures and dimensions in version control.
Tables and reports that count records read, written and quarantined.
Recovery steps and a workload cost owner.
Samples of the work
You should see what the work looks like before you hire anyone. These samples show the format we hand over. All values here are invented for illustration. They do not come from any patient, client or project.
One row for each field, including how nested data is handled.
| FHIR path | Silver column | Rule | Expectation |
|---|---|---|---|
| Observation.id, meta.versionId | observation_id, version_id | Sequence for AUTO CDC | Fail the update if the key is null |
| Observation.code.coding[] | loinc_code | Pick the LOINC system and keep the full array in bronze | Warn, and quarantine if no LOINC is found |
| Observation.valueQuantity | value, unit | Keep the source unit and convert in a separate column | Quarantine if the unit is unknown |
| Observation.subject.reference | patient_key | Resolve through the patient crosswalk | Quarantine if unresolved |
Counts from the event log and the quarantine table, tied together.
| Dataset | Rows read | Written to silver | Quarantined and top reason |
|---|---|---|---|
| fhir_observation | 120,400 | 120,112 | 288, unknown unit |
| fhir_encounter | 18,350 | 18,350 | 0 |
| claim_line_835 | 9,720 | 9,701 | 19, reversal without an original claim |
Features per Databricks documentation, October 2026. A reference design.
| Stage | What it holds |
|---|---|
| Sources | FHIR NDJSON, HL7 v2 via Kafka, X12 837 and 835, and CDC feeds. |
| Ingest | Auto Loader, Structured Streaming and Lakeflow pipelines. |
| Delta tables | Bronze for raw data, silver with expectations and AUTO CDC, a quarantine table, and dimensional gold. |
| Unity Catalog metric views | Measures, dimensions and materialization. |
| Consumers | Dashboards, Genie, Power BI, Tableau and MLflow models. |
| Across it all | Unity Catalog and the compliance security profile on each PHI workspace. |
| Step | What happens | Why it matters |
|---|---|---|
| $export writes NDJSON to storage | One file set per resource type. | |
| Auto Loader into bronze | String or VARIANT, with file and export ID. | Keeps full history. |
| Silver expectations | Warn, drop or fail. | Failures go to a quarantine table that you build. |
| AUTO CDC applies changes | A sequence column orders updates. | A late update cannot beat a newer one. |
| Gold dimensional tables | Row filters and column masks. | |
| Metric view | Dashboards and Genie read one definition. |
| Layer | Stores | Uses | Access | Promote when |
|---|---|---|---|---|
| Bronze | Raw FHIR, HL7 and X12 with full history | Auto Loader, Delta | Pipeline service principals | Delivery logged |
| Silver | Clean, non-aggregated records | Lakeflow, expectations, AUTO CDC | Engineers, approved analysts | Checks pass or quarantine |
| Gold | Dimensional models by domain | Materialized views, metric views | Analysts, with filters and masks | Reconciled and signed off |
Answer in this order. The first yes is usually your answer.
| Question | If yes |
|---|---|
| Already running one of them well? | Stay on it. |
| Large nested FHIR or HL7 data needing custom processing? | Databricks |
| Streaming or machine learning at the center of the work? | Databricks |
| SQL-first team focused on reporting? | Snowflake |
| Sharing governed data with partners? | Snowflake |
| None of these? | Test both on one workload. |
| Area | What applies |
|---|---|
| Catalogs | One each for dev, test and production. |
| Schemas | By domain: clinical, claims, reference. |
| Identities | Service principals for jobs, groups for people. |
| PHI controls | Row filters, column masks, lineage and audit. |
| You confirm | BAA, profile on each workspace, cloud BAA and keys. |
| Databricks | Encrypts PHI in the control plane. |
Test filters and masks on the real runtimes and access modes.
| Where | What happens there |
|---|---|
| Your environment | Holds your source systems, data platform and PHI, under your platform and cloud BAAs, your roles, your masks and your audit logs. |
| During the assessment | We use schemas, data dictionaries and synthetic samples. We do not use PHI or production credentials. |
| During a build | We sign a BAA with you first. Access uses approved roles, is logged and is the least needed. You grant, review and revoke it. |
| The code | Code and tests use synthetic data and live in your repository. |
Explore the implementation decisions
Explore ingestion, transformation and governance decisions for a healthcare lakehouse.
Define what raw, validated and consumer-ready layers are responsible for.
Explore event delivery and freshness requirements before committing to a streaming design.
Primary documentation for the standards and platform features named on this page.
Databricks Unity Catalog ↗Databricks medallion architecture ↗Delta Lake on Databricks ↗Databricks Auto Loader ↗Delta Lake change data feed ↗Lakeflow pipeline expectations (data quality) ↗Databricks HIPAA and the compliance security profile ↗FHIR Bulk Data Access IG ($export and NDJSON) ↗Unity Catalog metric views ↗Lakeflow AUTO CDC ↗Unity Catalog row filters and column masks ↗Databricks ai_forecast function ↗Databricks data profiling ↗Before we begin
Databricks supports HIPAA workloads: PHI must be processed in workspaces with the compliance security profile enabled and with a Databricks business associate agreement in place. Compliance also depends on how you configure access, logging and data handling, which is what we implement and document.
We request a FHIR Bulk Data $export, land the NDJSON files in cloud storage and ingest them with Auto Loader into bronze Delta tables, one per resource type. Silver tables flatten the nested JSON and apply validation, and incremental refreshes use _since plus resource versions to avoid duplicates.
It is a layered design: bronze keeps source data as received, silver holds validated and standardized records, and gold holds consumer-ready models such as patient cohorts, encounters or claims. In healthcare the value is traceability: a gold field can be followed back through silver to the original resource or message.
Yes. Unity Catalog gives one place for permissions, row filters, column masks, audit and lineage across workspaces. For PHI we separate identified and de-identified catalogs, run jobs as service principals and test access with the roles that will consume the data.
Yes. HL7 v2 messages arrive from an interface engine or Kafka topic and are parsed to segment and field tables. X12 837 and 835 files are parsed into claim, service line and adjustment tables with CARC and RARC codes preserved.
Databricks fits large nested FHIR and HL7 volumes, streaming and ML in one platform. Snowflake fits SQL-first analytics and secure data sharing. Many organizations run both; we define which platform owns each dataset.
Yes. We assess pipeline reliability, data models, permissions and workload costs, then scope targeted changes, including Hive metastore to Unity Catalog migration, rather than assuming a rebuild.
We key records on stable source identifiers and versions, then apply changes with AUTO CDC (formerly APPLY CHANGES) or MERGE so a corrected Observation or an adjusted claim replaces or versions the earlier row. Replay tests prove that reprocessing the same input does not double count.
Ask how they confirm the compliance security profile on each workspace, how failed records are quarantined when expectations only warn, drop or fail, and how a late change is kept from overwriting a newer one. Model development, if you need it, is scoped separately.
The assessment takes 2 to 3 weeks. We do not quote a build schedule before it, because source access and sign-off owners set the pace and differ widely. The costed build plan gives the sequence and its dependencies, once those are known.
The Healthcare Data Platform Assessment is a fixed $3,500. It takes 2 to 3 weeks and is credited against the build. It ends with a costed build plan. The build depends on sources, volume, freshness, governance and any rework in an existing workspace. Databricks and cloud usage are billed by those vendors.
Yes. We sign a business associate agreement with you when our work touches PHI, and we confirm the Databricks BAA and compliance security profile are in place before PHI enters the workspace.
Talk to our team
Tell us what you are planning, what exists today and what needs to change. We’ll review the context and discuss scope, dependencies and the next useful step.
hello@nirmitee.io →Explore Related Services
