Healthcare Data Engineering
We build the pipelines, models and controls that turn EHR, claims and device data into datasets your analytics, product and AI teams can rely on. Start with one reporting need or product workflow, then agree the source contracts, quality checks and operating responsibilities that make it dependable.
Scope a data readiness assessmentBook a 30-minute call with our data engineering lead
For healthtech leaders, data teams, providers and payers.
EHR extracts · claims · operational systems
Identity · meaning · provenance · freshness
Reviewed models · governed consumer access
Analytics · product features · AI development
Illustrative architecture · implementation boundaries agreed during scoping
The short answer
We ingest FHIR Bulk Data exports (NDJSON), HL7 v2 feeds, X12 837 and 835 files and database change streams. We land them in Databricks, Snowflake or your existing cloud warehouse, reconcile every load to its source and publish governed datasets with lineage, in identified or de-identified form. The first engagement ends with one reconciled data product in production, a runbook and named owners for the next source.
A focused first engagement
Use the first conversation to identify the consuming team, source systems and access dependencies. We then scope an assessment around the decisions your team needs to make.
Scope a data readiness assessment →We agree the assessment scope, fee and schedule after reviewing your context and access dependencies.
The problem behind the brief
Different identifiers, changing schemas and inconsistent definitions can survive a successful import. Start with the decision or product feature the data must support, then make quality and ownership explicit at every boundary.
What we build
Each workstream names the standard, the tool and the evidence your team reviews. Most engagements combine two or three of them around one consumer.
Clients for the FHIR Bulk Data Access $export operation at Patient, Group or system level. We poll the status endpoint, download NDJSON files per resource type and load them incrementally with _since, keeping resource versions and meta.lastUpdated.
Bulk FHIR export from Epic, Oracle Health and athenahealthADT, ORU, ORM and SIU feeds through Mirth Connect or a cloud converter, and X12 837, 835, 277CA and 999 files parsed into claim, line and adjustment tables with CARC and RARC codes preserved for denial analytics.
X12 EDI integration servicesLog-based CDC from EHR, practice-management and billing databases with Debezium, AWS DMS or native connectors, so inserts, updates and deletes arrive in order without nightly full extracts.
CDC for healthcare data warehousesBronze, silver and gold layers on Delta Lake or Snowflake, with patient, encounter, claim and member grain documented, plus OMOP CDM or FHIR-shaped analytics models where research or quality programs need them.
Databricks healthcare implementationHIPAA Safe Harbor removal of the 18 identifier types, or pipelines prepared for an Expert Determination, with date shifting, tokenized linkage keys and a re-identification risk review before data leaves the identified zone.
Safe Harbor vs Expert DeterminationCompleteness, validity, referential and terminology checks against LOINC, SNOMED CT, ICD-10-CM and RxNorm, with column-level lineage in Unity Catalog, OpenLineage or dbt docs so every published field traces back to a source.
Governance, lineage and PHI accessStart with your situation
The starting point
Analytics teams reconcile exports by hand and disagree on which records belong together.
Define source contracts, preserve provenance and build a reviewed model around an agreed patient, encounter or claim grain.
Illustrative project scenario, not a client case study.
Decisions we work through
Source totals reconcile to published datasets, with exclusions and uncertain matches explained.
Engineering scope
Agree the deliverables before implementation. Each workstream has a visible output, not just an activity list.
Choose a warehouse, lakehouse or combined approach around your workloads, existing cloud and operating model.
Build ingestion, incremental updates, mapping, reconciliation and recovery paths for the agreed sources.
Preserve provenance, units and terminology. Define identity matching and handling of late or corrected records with domain owners.
Define access, lineage, retention, monitoring and cost ownership with your security and data stakeholders.
Compare your options
We recommend the platform after the workload is clear. Most organizations already own one of these; the decision is usually what to add, not what to replace.
| Option | Best fit | Strength for healthcare data | Plan for |
|---|---|---|---|
| Databricks lakehouse | Large FHIR, HL7 and claims volumes, streaming, ML and AI workloads | Delta Lake, Unity Catalog lineage, Spark for nested FHIR JSON, MLflow | Compliance security profile and a Databricks BAA before PHI; cluster and job cost ownership |
| Snowflake warehouse | SQL-first analytics, reporting and partner data sharing | VARIANT for FHIR JSON, Secure Data Sharing, masking and row access policies | Business Critical edition and a Snowflake BAA before PHI; warehouse sizing |
| BigQuery with Google Cloud Healthcare API | Teams on Google Cloud with an operational FHIR store | Streaming export from FHIR stores to BigQuery, built-in de-identify operation | Schema of the exported FHIR tables and query cost controls |
| Operational FHIR store | Applications that read and write FHIR resources in real time | Standard API, SMART on FHIR, Bulk Data export for analytics | It is not an analytics engine; pair it with a warehouse for reporting |
| Existing SQL warehouse or Postgres | A bounded use case with modest volume | No new platform to buy or operate | Headroom for growth and a documented path if volume rises |
Platform subscriptions and cloud usage are contracted directly with the vendor. We scope the engineering and make those dependencies visible in the plan.
Related delivery
Public-source life-sciences data moves through collection, name reconciliation and approval before reporting and source-cited AI answers.
Delivery context: a public-data intelligence platform built with Frappe and Power BI, with AI assistants. The case shows source integration, reconciliation and review workflows.
Read the implementation case study →
The decisions underneath the delivery
Define the decision, report or product capability before selecting the platform. Agree data grain and freshness with the consuming team.
Retain source identifiers and history so transformations can be traced and reconciled. Document where a derived field changes meaning.
Agree matching rules, uncertain matches and review ownership. Do not silently merge records because two fields look similar.
Name the people responsible for rejected records, broken feeds and downstream changes. Include quality and freshness in service reviews.
Integration boundaries
Clear ownership keeps the first data product on schedule.
Your team or the EHR vendor grants export access, Bulk FHIR client registration, database credentials or SFTP drops. We prepare the technical request and test it.
Writing data back into an EHR or exchanging it with partners is interoperability work, scoped with our HL7, FHIR and X12 integration teams.
Your clinical and quality leads own measure definitions. We implement them as tested code and show every exclusion.
Databricks, Snowflake and cloud subscriptions and their BAAs sit between you and the vendor. We sign a BAA with you for our work with PHI.
Model training, clinical evaluation and serving are a separate stage built on the published datasets.
Explore the implementation decisions
Compare warehouse and lakehouse approaches against the workload, operating team and data consumers.
Work through source dependencies, platform responsibilities and the path to a usable data product.
Connect source fidelity, governed access and downstream consumption in an implementation architecture.
Start with context
Useful inputs for a focused first conversation. Please share project context, not patient records or credentials.
Your team takes forward
Choose the right foundation
They solve different problems. We scope the analytics and transformation layer separately from the endpoint your applications use to exchange FHIR resources.
FHIR data platform and FHIR server development →Healthcare Interoperability Services →Case study: a data foundation for governed AI →Primary documentation for the standards and platform features named on this page.
FHIR Bulk Data Access IG ($export and NDJSON) ↗45 CFR 164.514: HIPAA de-identification standard ↗HL7 US Core Implementation Guide ↗X12: healthcare transaction standards ↗HL7 FHIR R4 Provenance resource ↗OpenLineage specification ↗Databricks HIPAA and the compliance security profile ↗Snowflake editions: PHI support on Business Critical ↗From data to an AI workflow
Once the data path is dependable, scope one retrieval, extraction or assistant workflow around it. Preserve the source context and access rules through to the answer or proposed action.
Healthcare AI development and integration →Healthcare AI agent development →Delivery phases and timeline
Typical durations for one source-to-consumer data product. Source access is the most common schedule risk, so we start it in week one.
2 to 3 weeks. Inventory sources, access paths, consumers, freshness and quality risks; choose the platform boundary.
6 to 10 weeks. Build ingestion, models and quality checks for one source and one consumer.
2 to 4 weeks. Exercise replay, late data and failure paths; set alerts, owners and lineage.
3 to 6 weeks per additional source, on a rolling backlog.
Cost drivers
We estimate after the assessment. These are the factors that move the estimate most.
A Bulk FHIR endpoint, an HL7 feed, an X12 file drop and a database CDC stream each need different tooling and tests.
Years of history to backfill, resource counts and daily change volume shape both engineering and platform usage.
Daily batch is simpler than hourly micro-batch, which is simpler than streaming with ordering guarantees.
Cross-source patient matching and local code mapping to LOINC, SNOMED CT or RxNorm add reviewed rules and test sets.
Safe Harbor is rule-based. Expert Determination adds a qualified statistician and documented risk analysis.
Separate dev, test and production catalogs, access reviews and lineage requirements add setup and evidence work.
Before we begin
They are the engineering work that moves clinical, claims and operational data from source systems into governed datasets: ingestion from FHIR, HL7 v2, X12 and databases, transformation into documented models, quality and reconciliation checks, de-identification, lineage and the operating runbook. The output is data your analysts, product teams and AI workflows can use without re-checking it by hand.
A healthcare data engineer builds and runs the pipelines between EHR, claims and device sources and the teams that consume the data. The healthcare part is the standards and rules: FHIR resources and Bulk Data export, HL7 v2 segments, X12 837 and 835 loops, terminology such as LOINC and ICD-10-CM, patient matching and HIPAA minimum-necessary access.
Not automatically. Databricks suits large nested FHIR and HL7 volumes, streaming and ML. Snowflake suits SQL-first analytics and partner data sharing. A modest use case can run on your existing warehouse. We recommend after reviewing the workload, existing investments, skills and operating cost.
We use the access path each EHR supports for your organization: FHIR Bulk Data $export where it is enabled, HL7 v2 feeds through an interface engine, vendor reporting databases and extracts, or scheduled files. Your organization authorizes the access; we build and test the client, the load and the reconciliation.
FHIR Bulk Data Access defines the $export operation, which returns a set of files for a patient group or the whole system. Each file is newline-delimited JSON (NDJSON), one FHIR resource per line, grouped by resource type. It suits population-level loads into a lakehouse or warehouse, with _since for incremental refreshes.
Identified data stays in an access-controlled zone with audit logging. For de-identified datasets we apply HIPAA Safe Harbor, removing the 18 identifier types, or build the pipeline an expert needs for an Expert Determination. We sign a BAA with you for work involving PHI and confirm your platform BAAs are in place.
Interoperability exchanges data between systems, often in real time and often back into the EHR. Data engineering prepares and operates datasets for analytics, reporting and product workloads. Many projects need both, so we define the boundary and connect the two scopes.
Every load is reconciled to the source: record counts, control totals such as billed and paid amounts, referential checks between patients, encounters and claims, and terminology validity. Exceptions go to an owned queue with a reason code, so a published dataset states exactly what it excludes.
Yes. We publish reproducible, versioned datasets with provenance, access boundaries and quality checks, which is what retrieval, extraction and model training need. Model training, clinical evaluation and serving are scoped as a separate stage on top of that foundation.
An assessment takes 2 to 3 weeks. A first reconciled data product for one source and one consumer typically takes 6 to 10 weeks, followed by 2 to 4 weeks of hardening and handover. Each additional source usually takes 3 to 6 weeks.
The number and type of sources, history and volume, freshness requirements, identity and terminology work, the de-identification method and governance requirements. We give an estimate after the assessment, and platform charges are billed by the platform vendor.
Yes. We sign a business associate agreement when our work touches PHI. Nirmitee is ISO 27001:2022 certified, and we start discovery and test design with synthetic or de-identified examples before any production data is involved.
Talk to our team
Tell us what you are planning, what exists today and what needs to change. We’ll review the context and discuss scope, dependencies and the next useful step.
hello@nirmitee.io →Explore Related Services
