Healthcare Data Pipelines
When a data feed fails quietly, people act on wrong or late numbers and nobody knows until someone complains. Our healthcare data pipeline services move your clinical and claims data on time, hold back anything that fails a check and show what happened. You start with a fixed-price assessment and see the plan and the cost before you commit to a build.
Book the Healthcare Data Platform AssessmentFor teams facing unreliable feeds, manual exports or inconsistent datasets.
Data arrives late, twice or not at all, and people find out from a wrong report.
Data that arrives on a schedule you can see, with failed checks holding the release.
Release waits on a reconciliation check, and the original data is kept for a rerun.
$3,500 fixed for the assessment. 2 to 3 weeks, credited against the build.
Book the assessment and get a costed plan you can use with any vendor.
The business case
A pipeline is only worth paying for when people can rely on what comes out of it.
Healthcare data arrives from many places, on different schedules, and it changes after it arrives. Lab results get corrected and claims get reversed. Without a controlled route, a missed file or a resent message quietly changes your numbers.
Our view
A green job status tells you the code ran. It does not tell you the data is complete or correct. We treat a healthcare pipeline as finished when its output ties back to the source and your team can replay a bad day without double counting.
Streaming adds ordering, replay and on-call work. Use it when someone acts on the event within minutes, as with ADT-driven bed management. Remittance files arrive in batches, and streaming cannot make them earlier.
A corrected ORU, a reversed 835 and a merged patient are everyday events in healthcare. Design for them from the start, since retrofitting them means rebuilding the keys.
Without an untouched copy and a delivery ledger, you cannot prove what the source sent or rebuild after a bad mapping.
Mirth Connect is good at routing and acknowledging messages. Analytics history, reconciliation and modeling belong downstream of it.
When we would tell you not to build a new pipeline: an existing one fails for a known reason, such as one mapping or missing retries. Repair it first and spend the money on the next source.
For decision makers
These are the questions that decide whether a data engineering vendor is safe to hire. Here are our answers, and how to test them.
| Risk | How we control it |
|---|---|
| A source changes without notice | Contract checks at intake that quarantine and alert |
| Replay creates duplicates | Idempotent writes, tested before release |
| PHI appears in logs or dead-letter queues | Restrict and retain these stores like production data, and log identifiers without payloads |
| Streaming costs more than it returns | Require a named person who acts on the fresher data |
| One person holds all the knowledge | Rehearse the runbook with your team before handover |
The first step
Fixed price: $3,500. 2 to 3 weeks, credited against the build.
A fixed-price first step for one decision you need the data to support. You finish with enough evidence to approve, change or stop the build, and the outputs are yours to use with any vendor.
Book the Healthcare Data Platform AssessmentThe assessment is complete when all five outputs are delivered and walked through with your owners. Sending the form starts a conversation, and you pay nothing until you agree the assessment.
How the work runs
We reproduce failures before designing fixes, and test the bad days on purpose before release.
Follow one record from source to report, and reproduce any current failure.
You review: A failure map with evidence
Agree schema, keys, update behavior, delivery windows and quality rules for each source.
You review: A source contract
Implement intake, the ledger, validation, transformation and exception handling.
You review: Versioned code with tests
Feed in duplicates, late records, corrections, rejected records and a full replay.
You review: Exercise results you can inspect
Tie counts and control totals to the source.
You review: A reconciliation report
Set freshness and volume alerts, assign dead-letter ownership and walk through recovery.
You review: A runbook and named owners
Related delivery
Read how Nirmitee diagnosed and hardened an existing clearinghouse X12 pipeline inside a live behavioral-health EHR, using reviewed fixes and automated tests.
Delivery context: an operational claims integration. Its source and workflow constraints shape how we design data pipelines.
Read the implementation case study →
Cost drivers
Pipeline effort follows source behavior more than data volume.
Each HL7 feed, FHIR endpoint, X12 trading partner or database adds its own contract and tests.
Streaming and CDC need ordering, idempotency and back-pressure handling that batch does not.
Corrections, cancellations, merges and deletes require versioned keys and restatement logic.
Missing identifiers, local codes and free text add mapping and review work.
Repairing undocumented jobs starts with reverse engineering the current behavior.
On-call coverage, alert routing and SLAs add operational setup.
For your engineers
What your technical team will want to check: sources, batch or streaming choices, layer rules, failure handling and samples of the work.
The short answer
We build and repair the pipelines that move healthcare data from source to consumer: HL7 v2 ADT and ORU feeds, FHIR Bulk Data NDJSON and FHIR Subscriptions, X12 837, 835 and 277CA files, flat-file drops and log-based CDC from EHR and billing databases. Each pipeline has a source contract, validation with a quarantine path, reconciliation to the source, safe replay and freshness alerts routed to a named owner.
What we build
Pipelines named by source and standard, each with the behavior it must handle.
ADT, ORU, ORM, SIU and MDM messages from Mirth Connect or another interface engine, over MLLP or a queue, parsed to segment-level tables with message control IDs used for de-duplication and ordering.
HL7 integration servicesBulk Data $export clients for population loads with NDJSON per resource type, plus FHIR Subscriptions or change feeds for near-real-time updates.
FHIR subscriptions and event-driven pipelines837 claim, 835 remittance, 277CA and 999 acknowledgment files parsed to claim, line and adjustment tables, with CARC, RARC and claim-status codes kept for denial and payment analytics.
X12 837 and 835 to FHIRLog-based CDC with Debezium, AWS DMS or native connectors from EHR, practice-management and billing databases, with ordered inserts, updates and deletes and a replayable offset.
CDC for healthcare data warehousesLocal code mapping to LOINC, SNOMED CT, RxNorm and ICD-10-CM, and patient matching rules with review queues for uncertain matches.
Terminology mapping in productionCheckpoints, idempotent writes, dead-letter queues, freshness and volume alerts, and OpenTelemetry traces across interface engine, pipeline and warehouse.
Instrumenting HL7 and FHIR pipelinesSources handled
The intake method follows from how the source delivers data. These are the patterns we use.
| Source | Intake pattern | Acknowledgment or completeness signal | Common failure |
|---|---|---|---|
| HL7 v2 ADT, ORU, ORM, SIU | Mirth Connect or another engine over MLLP, then a message topic or store | ACK per message and a sequence check | Resends, and A08 updates arriving before A01 admits |
| FHIR Bulk Data | Kick off $export, poll status, download NDJSON and load with _since | The manifest of output files | Expired download links and partial downloads |
| FHIR Subscriptions | Webhook intake with a queue behind it | Event numbers and a catch-up query | Missed events during an outage |
| X12 837, 835, 277CA, 999 | SFTP or clearinghouse API, then a loop-aware parser | 999 and 277CA responses, and 835 control totals | Reversals counted as new payments |
| Databases | Log-based CDC with Debezium or AWS DMS | Log position and a periodic count comparison | Schema changes that stop the connector |
| Flat files and vendor exports | Scheduled pickup with a layout check | Expected-file ledger and row counts | A changed layout loading the wrong columns |
Integration boundaries
Explicit ownership keeps failures from falling between teams.
Grant access, publish schema changes in advance and own corrections at the source.
If Mirth Connect or another engine feeds the pipeline, channel changes are coordinated with its owner or our Mirth team.
We own ingestion code, validation, reconciliation, replay and the runbook for the agreed scope.
Consuming teams define freshness and completeness expectations and accept the published dataset.
Sending data back into an EHR is integration work, scoped with our interoperability team.
Compare your options
Choose the pattern from the freshness your consumer needs, not from a default real-time promise.
| Pattern | Typical freshness | Good for | Engineering watch-points |
|---|---|---|---|
| Scheduled file batch (SFTP, CSV, CCD) | Daily or hourly | Payer files, vendor extracts, legacy systems | Late or partial files, re-sent files, schema drift |
| FHIR Bulk Data $export | Daily or weekly, incremental with _since | Population analytics, quality measures, payer data exchange | Export job limits, deleted resources, version handling |
| Log-based CDC | Seconds to minutes | EHR and billing databases you are permitted to read | Schema changes, ordering, initial snapshot, source load |
| Event streaming (HL7 v2 or FHIR Subscriptions to Kafka) | Seconds | ADT alerts, lab results, care coordination | Duplicates, out-of-order events, consumer back-pressure |
Tooling follows the pattern and your stack: Mirth Connect, Kafka, Airflow, dbt, Spark, Databricks, Snowflake or your cloud services.
Medallion layers
Whatever you call the layers, a pipeline needs these boundaries. They are what make a replay safe.
| Layer | Pipeline rule | What it makes possible |
|---|---|---|
| Bronze: landing | Append only, with source, delivery ID and received time on every record | Rebuild everything downstream after a bad mapping |
| Silver: validated | Idempotent writes keyed on source ID and version, with rejected records kept and given a reason | Rerun without creating duplicates |
| Gold: published | Released only after reconciliation passes | Release waits until the day's load reconciles |
Dimension mapping
Do the mapping once, in one place, and keep the original value. Changing a rule then means a rerun and leaves the raw data untouched.
| Dimension | Where it comes from | Pipeline treatment |
|---|---|---|
| Patient | PID segment, Patient resource, 837 subscriber loop | Keep source identifiers side by side and resolve them in a separate, rerunnable step |
| Provider | PV1, Practitioner, NM1 loops | Normalize to NPI where present, with the role kept |
| Payer | Clearinghouse IDs, plan files | Crosswalk table with an owner and effective dates |
| Lab codes | OBX-3 local codes | Crosswalk to LOINC, with unmapped codes sent to a work queue |
| Time | MSH, event and service timestamps | Store in UTC with the source time zone and name each date role |
Semantic models
A pipeline change can quietly alter a metric. So we tie each metric to the pipeline rules it depends on, and test them together.
| Metric | Pipeline rule it depends on | What breaks it |
|---|---|---|
| Net paid amount | Apply 835 reversals in sequence, by claim control number | Loading a reversal as a new payment |
| Current admitted patients | Process ADT events in event order, with cancels applied | A cancelled admission that stays open |
| Abnormal result count | Keep only the latest result version for each order | Counting preliminary and final results together |
| Eligible population | Apply effective dates from the 834 or member file | Overwriting past coverage with today's status |
Refresh frequency
Ask how soon someone acts on the data, then choose the simplest method that meets that need.
| Method | Fits | You take on | Avoid when |
|---|---|---|---|
| Scheduled batch | Claims, remittances, quality reporting and monthly close | Delivery windows and backfills | Someone must act within the hour |
| Micro-batch | Lab trends and operational reporting during the day | Late-arriving data handling | The source only delivers daily |
| Log-based CDC | Billing and practice databases, including deletes | Log access, schema changes and connector upkeep | The vendor will not allow log access |
| Streaming | ADT-driven workflows and alerts | Ordering, replay and on-call cover | Nobody acts on the event in minutes |
Cleanup problems and fixes
These are the failure modes we test for on purpose.
| Failure | Fix | Test that proves it |
|---|---|---|
| Same message delivered twice | Idempotent write on message control ID | Replay a day and compare totals |
| Events out of order | Sequence on event time and version, and hold or re-apply late events | Shuffle a test batch and compare the end state |
| Corrected result or claim | Supersede the prior version and keep history | Send a correction and check the metric moves once |
| Source schema changes | Contract check at intake that quarantines unknown shapes | Add a column and rename another in test |
| Partial file or export | Compare with manifest and trailer counts before processing | Truncate a file in test |
| Bad records pile up unseen | A dead-letter queue with an owner, reason codes and an age alert | Inject bad records and confirm the alert |
Dashboards people act on
Pipeline monitoring needs its own screens. These are the views we set up with the operations team.
| Who opens it | The question | The action it triggers |
|---|---|---|
| Data on-call | What delivery is missing or late? | Call the source owner or start a replay |
| Data steward | What is in the dead-letter queue, and how old is it? | Fix the mapping or reject with a reason |
| Analytics lead | Did today's load reconcile? | Hold or release the affected dashboards |
| Platform owner | Which pipelines cost the most for the decisions they serve? | Lower the refresh rate or retire the job |
Forecasting loop
The first useful forecast is about the pipeline itself: how much data should arrive. A gap against that forecast catches trouble no error log will show.
Record normal volume by source, weekday and hour.
Set an expected range for each delivery.
Flag deliveries outside the range even when the job itself succeeded.
Label each alert as a real issue or a false alarm.
Adjust ranges, and feed verified history to downstream forecasts.
Business forecasts can use the same reconciled history once the pipeline has been stable long enough to trust.
Deliverables
All of it lives in your repository and your accounts.
Versioned, tested and deployable by your team.
One for each feed, kept with the code.
What was expected, received, rejected and released.
Duplicate, late, corrected, partial and replay scenarios using synthetic data.
Freshness, volume and age alerts, each with a named owner.
How to replay, backfill and recover, rehearsed with your team.
Samples of the work
You should see what the work looks like before you hire anyone. These samples show the format we hand over. All values here are invented for illustration. They do not come from any patient, client or project.
The ledger shows what was expected and what happened to it, even when no job failed.
| Delivery | Expected | Received | Status and action |
|---|---|---|---|
| ADT feed, site A | Within normal volume range | In range | Released |
| 835 file, clearinghouse | One file by morning | Not received | Alert sent to the source owner, dashboards held |
| FHIR export, Observation | All files in the manifest | 11 of 12 | Partial: reload the missing file, then process |
One reversal followed through to the metric, so the total moves once.
| Step | What arrives or happens | Net paid for this claim |
|---|---|---|
| First 835 | Claim paid 120.00 | 120.00 |
| Reversal in a later 835 | Status 22, minus 120.00, same claim control number | 0.00 |
| Corrected payment | Claim paid 95.00 | 95.00 |
| Replay of all three files | Each file is already in the ledger, so it is skipped | 95.00 |
A replay should change nothing except what was wrong.
| Test | Before | After replay | Result |
|---|---|---|---|
| Rows in validated lab table | 48,210 | 48,210 | Pass: no duplicates |
| Corrected results applied once | 6 corrections sent | 6 versions replaced | Pass |
| Net paid against control total | Equal | Equal | Pass |
| Dead-letter records with a reason | 19 of 19 | 19 of 19 | Pass |
| Stage | What it holds |
|---|---|
| Sources | HL7 v2 over MLLP, FHIR $export, FHIR Subscriptions, X12 837 and 835, database logs and flat files. |
| Intake and delivery ledger | Mirth or another engine, NDJSON loader, SFTP pickup, Debezium or DMS, with expected deliveries checked against what was received. |
| Raw store, append only | Delivery ID, received time and original payload. |
| Validate and transform | Contract check, idempotent write, dead-letter queue and code crosswalks. |
| Reconcile, then release | Control totals, freshness and consumers. |
| Alerts | Missing delivery, odd volume and aged dead letters. |
| Step | What is checked | What happens |
|---|---|---|
| Message arrives | Checked against the ledger. | |
| Seen this control ID? | The exact same message, resent. | Yes: skip it and count it. |
| Passes the contract? | Schema, keys and codes. | No: dead-letter queue, with a reason. |
| Newer version of a record? | A corrected result or a reversal. | Yes: replaces the prior version and keeps history. |
| Write to validated table | Keyed on source ID and version. | |
| Totals tie to source? | Counts and control totals. | No: release held. |
| Layer | Rule | Keeps | Allows |
|---|---|---|---|
| Bronze: landing | Append only | Payload, delivery ID, received time | Full rebuild after a bad mapping |
| Silver: validated | Idempotent, keyed on ID and version | Rejected records, with a reason | Rerun with no duplicates |
| Gold: published | Release only after reconciliation | Grain and metric owner | Consumers get reconciled days only |
Answer in this order. The first yes is the method to use.
| Question | If yes |
|---|---|
| Does someone act on the event within minutes? | Streaming |
| Database source, log access allowed, deletes matter? | Log-based CDC |
| Source delivers through the day, and the same-day number is used? | Micro-batch |
| None of these? | Scheduled batch |
| Who | What they do |
|---|---|
| Alert fires | Missing delivery, odd volume or aged dead letters. |
| On-call engineer: triage | Is it the source, the pipeline or the mapping? |
| Source owner | Resends or fixes the feed. |
| Data steward | Clears dead letters: fix the mapping or reject. |
| On-call engineer: replay | Replays from raw, then checks totals. |
| Analytics lead | Releases or holds affected dashboards. |
| Where | What happens there |
|---|---|
| Your environment | Holds your source systems, data platform and PHI, under your platform and cloud BAAs, your roles, your masks and your audit logs. |
| During the assessment | We use schemas, data dictionaries and synthetic samples. We do not use PHI or production credentials. |
| During a build | We sign a BAA with you first. Access uses approved roles, is logged and is the least needed. You grant, review and revoke it. |
| The code | Code and tests use synthetic data and live in your repository. |
Explore the implementation decisions
Work through data preparation, quality and downstream use.
Explore event delivery and freshness requirements before committing to a streaming design.
Examine source notifications and the behavior a downstream consumer must handle.
Primary documentation for the standards and platform features named on this page.
FHIR Bulk Data Access IG ($export and NDJSON) ↗X12: healthcare transaction standards ↗HL7 US Core Implementation Guide ↗Debezium change data capture ↗OpenLineage specification ↗HL7 FHIR R4 Provenance resource ↗45 CFR 164.514: HIPAA de-identification standard ↗Databricks medallion architecture ↗Before we begin
It is the automated path that receives data from a source such as an EHR, lab, clearinghouse or device platform, validates and transforms it, and publishes it for a consumer such as a warehouse, product feature or AI workflow. A production pipeline also reconciles to the source, quarantines bad records and recovers from failure.
Most modern healthcare warehouses use ELT: land the source as received, then transform in the warehouse so raw data stays available for audit and reprocessing. ETL still fits when data must be filtered or de-identified before it reaches the destination.
Yes. We receive HL7 v2 messages through Mirth Connect or another interface engine, parse segments such as PID, PV1, OBR and OBX into tables, de-duplicate on message control ID and preserve the raw message for audit.
We parse 835 files into payment, claim and service line tables, keep CAS adjustment groups with CARC and RARC codes, and reconcile totals to the BPR payment amount. The result supports denial analytics and payment posting checks.
Change data capture reads a database transaction log so inserts, updates and deletes flow downstream in order, without nightly full extracts. It fits when you are permitted to read an EHR or billing database and need fresher data than batch provides.
We key records on stable source identifiers and versions, apply corrections as updates rather than new rows, and allow late data within an agreed window. Consumers see a restatement flag when historical numbers change.
Ask them to show a replay test, what happens to a corrected result, where rejected records go and who owns them. Ask too what signals a missing delivery when no job has failed. We work on Databricks, Snowflake and your existing warehouse, so the tool should follow from those answers.
Yes. We start with failure evidence, source behavior and operational ownership, then deliver targeted remediation with measurable acceptance checks.
We track freshness, volume, rejected-record counts and reconciliation results per run, route alerts to a named owner and trace records across interface engine, pipeline and warehouse. Dead-letter queues keep failed records for review and replay.
No. Discovery and test design start with synthetic or de-identified examples. Later access to production data happens in approved environments under a BAA.
The assessment takes 2 to 3 weeks. We do not quote a build schedule before it, because source access and sign-off owners set the pace and differ widely. The costed build plan gives the sequence and its dependencies, once those are known.
The Healthcare Data Platform Assessment is a fixed $3,500. It takes 2 to 3 weeks and is credited against the build. It ends with a costed build plan. The build depends on source count and variety, freshness, update behavior, source data quality and the state of any existing pipeline.
Talk to our team
Tell us what you are planning, what exists today and what needs to change. We’ll review the context and discuss scope, dependencies and the next useful step.
hello@nirmitee.io →Explore Related Services
