HL7 v2 feed pipelines
ADT, ORU, ORM, SIU and MDM messages from Mirth Connect or another interface engine, over MLLP or a queue, parsed to segment-level tables with message control IDs used for de-duplication and ordering.
HL7 integration servicesHealthcare Data Pipelines
Healthcare Data Pipeline Services
Resolve unreliable feeds, repeated manual exports and unexplained differences between source and output. We design ingestion, validation and recovery around the real update behavior of your EHR, claims and operational systems, from HL7 v2 feeds to FHIR Bulk Data and X12 835 files.
Scope a healthcare pipeline reviewBook a 30-minute call with our data engineering lead
For teams facing unreliable feeds, manual exports or inconsistent datasets.
Engineering blueprintData → decisions
- 01Receive
Source identity + batch or event context
- 02Validate
Accept · reject · quarantine with a reason
- 03Transform and publish
Deterministic mapping + reconciliation
- 04Recover
Checkpoint · retry · replay · alert an owner
Illustrative architecture · implementation boundaries agreed during scoping
Evidence you can check
- ISO 27001:2022 certified
- HIPAA-enabled: we sign BAAs
- Open-source X12 clearinghouse simulator: 270/271, 278, 837, 999/277CA, 835, 276/277
- Our team has built 350+ interfaces on Mirth Connect
- 18 open-source healthcare repositories on GitHub
The short answer
What a healthcare data pipeline service delivers
We build and repair the pipelines that move healthcare data from source to consumer: HL7 v2 ADT and ORU feeds, FHIR Bulk Data NDJSON and FHIR Subscriptions, X12 837, 835 and 277CA files, flat-file drops and log-based CDC from EHR and billing databases. Each pipeline has a source contract, validation with a quarantine path, reconciliation to the source, safe replay and freshness alerts routed to a named owner.
- Who it is for
- Teams facing unreliable feeds, manual exports, duplicate records or numbers that do not match the source.
- What we build
- Batch, CDC and streaming pipelines with quality rules, reconciliation, replay and monitoring.
- The outcome
- Data that arrives on time, matches the source and recovers from failure without manual clean-up.
A focused first engagement
Turn a recurring pipeline failure into a testable fix.
Bring one failing data path, its expected behavior and the team that depends on it. We scope the investigation around source changes, rejected records, freshness and recovery.
Scope a healthcare pipeline review →Assessment deliverables
- Source contract and failure-path map
- Quality and reconciliation rules
- Retry, replay and recovery acceptance tests
- A remediation scope with operational ownership
We agree the assessment scope, fee and schedule after reviewing your context and access dependencies.
The problem behind the brief
The difficult records need a designed path, too.
Late events, corrected results, duplicate deliveries and schema changes are normal integration conditions. We turn them into explicit behaviors and acceptance tests rather than manual clean-up after each load.
What we build
Healthcare data pipelines we build
Pipelines named by source and standard, each with the behavior it must handle.
FHIR bulk and event pipelines
Bulk Data $export clients for population loads with NDJSON per resource type, plus FHIR Subscriptions or change feeds for near-real-time updates.
FHIR subscriptions and event-driven pipelinesX12 claims pipelines
837 claim, 835 remittance, 277CA and 999 acknowledgment files parsed to claim, line and adjustment tables, with CARC, RARC and claim-status codes kept for denial and payment analytics.
X12 837 and 835 to FHIRChange data capture
Log-based CDC with Debezium, AWS DMS or native connectors from EHR, practice-management and billing databases, with ordered inserts, updates and deletes and a replayable offset.
CDC for healthcare data warehousesTerminology and identity steps
Local code mapping to LOINC, SNOMED CT, RxNorm and ICD-10-CM, and patient matching rules with review queues for uncertain matches.
Terminology mapping in productionObservability and recovery
Checkpoints, idempotent writes, dead-letter queues, freshness and volume alerts, and OpenTelemetry traces across interface engine, pipeline and warehouse.
Instrumenting HL7 and FHIR pipelinesStart with your situation
What kind of pipeline problem are you solving?
The starting point
Replace manual data exports
Someone has to download, combine and upload files before the next team can work.
Document the manual rules, agree a source contract and automate the path with validation and an explicit exception queue.
Illustrative project scenario, not a client case study.
Decisions we work through
- File completeness and arrival expectations
- Manual rules hidden in spreadsheets
- Failed-load ownership
Example acceptance check
Scope a healthcare pipeline review →The agreed input is processed without manual intervention, while malformed examples follow a visible exception path.
Engineering scope
What we work on.
What you can review.
Agree the deliverables before implementation. Each workstream has a visible output, not just an activity list.
Source contracts
Define schemas, identifiers, update semantics, delivery windows and permitted data use with each source owner.
ETL / ELT implementation
Choose batch, incremental or event-driven processing for the real freshness requirement, not a default real-time promise.
Quality & reconciliation
Test completeness, duplicates, terminology, units and accepted exceptions. Route rejected records to an owned resolution process.
Recovery & observability
Implement safe retries, checkpoints, replay and alerting. Agree who responds when freshness or quality expectations are missed.
Compare your options
Batch, CDC and streaming healthcare pipelines compared
Choose the pattern from the freshness your consumer needs, not from a default real-time promise.
| Pattern | Typical freshness | Good for | Engineering watch-points |
|---|---|---|---|
| Scheduled file batch (SFTP, CSV, CCD) | Daily or hourly | Payer files, vendor extracts, legacy systems | Late or partial files, re-sent files, schema drift |
| FHIR Bulk Data $export | Daily or weekly, incremental with _since | Population analytics, quality measures, payer data exchange | Export job limits, deleted resources, version handling |
| Log-based CDC | Seconds to minutes | EHR and billing databases you are permitted to read | Schema changes, ordering, initial snapshot, source load |
| Event streaming (HL7 v2 or FHIR Subscriptions to Kafka) | Seconds | ADT alerts, lab results, care coordination | Duplicates, out-of-order events, consumer back-pressure |
Tooling follows the pattern and your stack: Mirth Connect, Kafka, Airflow, dbt, Spark, Databricks, Snowflake or your cloud services.
Related delivery
Clearinghouse claims integration in a behavioral-health EHR
Read how Nirmitee diagnosed and hardened an existing clearinghouse X12 pipeline inside a live behavioral-health EHR, using reviewed fixes and automated tests.
Delivery context: an operational claims integration. Its source and workflow constraints shape how we design data pipelines.
Read the implementation case study →
The decisions underneath the delivery
Every record needs a path, including the difficult ones.
Delivery semantics
Document duplicate delivery, out-of-order records, retry behavior and the stable keys available from the source.
Schema change
Decide which changes are compatible, which should stop a pipeline and how the source owner is notified.
Clinical context
Review units, codes, timestamps and corrected results. Preserve source meaning when the target cannot represent every field.
Recovery boundaries
Identify what can be replayed safely, how consumers recognize corrections and how the team verifies the repaired output.
Integration boundaries
Integration boundaries for a healthcare pipeline project
Explicit ownership keeps failures from falling between teams.
Source owners
Grant access, publish schema changes in advance and own corrections at the source.
Interface engine
If Mirth Connect or another engine feeds the pipeline, channel changes are coordinated with its owner or our Mirth team.
Pipeline and quality
We own ingestion code, validation, reconciliation, replay and the runbook for the agreed scope.
Consumers
Consuming teams define freshness and completeness expectations and accept the published dataset.
Real-time write-back
Sending data back into an EHR is integration work, scoped with our interoperability team.
Explore the implementation decisions
Guides for the work ahead.
FHIR data pipelines for AI and analytics
Work through data preparation, quality and downstream use.
Streaming healthcare data with Kafka and FHIR
Explore event delivery and freshness requirements before committing to a streaming design.
FHIR subscriptions and event-driven pipelines
Examine source notifications and the behavior a downstream consumer must handle.
Start with context
Bring the constraints.
We’ll help shape the scope.
Useful inputs for a focused first conversation. Please share project context, not patient records or credentials.
- Representative synthetic source examples
- Update, correction and deletion behavior
- Expected volume and freshness
- Target platform and operational owner
Your team takes forward
Work that stays useful
after the engagement.
- 01Explicit data contracts and mappings
- 02Automated pipeline and quality tests
- 03Replay and recovery procedures
- 04Freshness monitoring and issue ownership
Choose the right foundation
Analytics platform?
Operational FHIR store?
They solve different problems. We scope the analytics and transformation layer separately from the endpoint your applications use to exchange FHIR resources.
FHIR data platform and FHIR server development →Healthcare Interoperability Services →Case study: a data foundation for governed AI →Standards and platform references
Primary documentation for the standards and platform features named on this page.
FHIR Bulk Data Access IG ($export and NDJSON) ↗X12: healthcare transaction standards ↗HL7 US Core Implementation Guide ↗Debezium change data capture ↗OpenLineage specification ↗HL7 FHIR R4 Provenance resource ↗45 CFR 164.514: HIPAA de-identification standard ↗From data to an AI workflow
Keep AI inputs current, traceable and recoverable.
A retrieval or extraction workflow inherits the behavior of its data pipeline. Make stale inputs, source changes and failed refreshes visible before building on the output.
Healthcare AI development and integration →Healthcare AI agent development →Define the next delivery boundary
- Track freshness and the source version used downstream
- Define what an AI workflow does when data is missing or stale
- Re-run quality and evaluation checks when source behavior changes
Delivery phases and timeline
Healthcare data pipeline project timeline
Typical durations for a diagnostic, a repair or a new source.
Diagnostic
1 to 2 weeks. Reproduce the failure, map source behavior and agree acceptance checks.
Source contract and design
1 to 2 weeks. Schema, keys, update semantics, delivery windows and quality rules per source.
Build and exercise
3 to 6 weeks per source. Implement, then test duplicates, late, corrected, rejected and replayed records.
Monitor and hand over
1 to 2 weeks. Freshness and volume alerts, dead-letter ownership and runbook walkthrough.
Cost drivers
What drives the cost of a healthcare data pipeline
Pipeline effort follows source behavior more than data volume.
Source count and variety
Each HL7 feed, FHIR endpoint, X12 trading partner or database adds its own contract and tests.
Freshness
Streaming and CDC need ordering, idempotency and back-pressure handling that batch does not.
Update semantics
Corrections, cancellations, merges and deletes require versioned keys and restatement logic.
Data quality at the source
Missing identifiers, local codes and free text add mapping and review work.
Existing pipeline state
Repairing undocumented jobs starts with reverse engineering the current behavior.
Monitoring expectations
On-call coverage, alert routing and SLAs add operational setup.
Before we begin
Healthcare data pipeline questions buyers ask
What is a healthcare data pipeline?
It is the automated path that receives data from a source such as an EHR, lab, clearinghouse or device platform, validates and transforms it, and publishes it for a consumer such as a warehouse, product feature or AI workflow. A production pipeline also reconciles to the source, quarantines bad records and recovers from failure.
Should healthcare data use ETL or ELT?
Most modern healthcare warehouses use ELT: land the source as received, then transform in the warehouse so raw data stays available for audit and reprocessing. ETL still fits when data must be filtered or de-identified before it reaches the destination.
Can you build pipelines from HL7 v2 feeds?
Yes. We receive HL7 v2 messages through Mirth Connect or another interface engine, parse segments such as PID, PV1, OBR and OBX into tables, de-duplicate on message control ID and preserve the raw message for audit.
How do you process X12 835 remittance files?
We parse 835 files into payment, claim and service line tables, keep CAS adjustment groups with CARC and RARC codes, and reconcile totals to the BPR payment amount. The result supports denial analytics and payment posting checks.
What is change data capture and when do we need it?
Change data capture reads a database transaction log so inserts, updates and deletes flow downstream in order, without nightly full extracts. It fits when you are permitted to read an EHR or billing database and need fresher data than batch provides.
How do you handle corrected lab results and late-arriving data?
We key records on stable source identifiers and versions, apply corrections as updates rather than new rows, and allow late data within an agreed window. Consumers see a restatement flag when historical numbers change.
Do you only use Databricks or Snowflake?
No. We build pipelines within your existing cloud and data stack, including Airflow, dbt, Kafka, cloud-native services and Postgres. The tool choice follows your requirements, maintainability and commercial constraints.
Can you fix a pipeline that already exists?
Yes. We start with failure evidence, source behavior and operational ownership, then deliver targeted remediation with measurable acceptance checks.
How do you monitor healthcare pipelines?
We track freshness, volume, rejected-record counts and reconciliation results per run, route alerts to a named owner and trace records across interface engine, pipeline and warehouse. Dead-letter queues keep failed records for review and replay.
Do you need real patient data to start?
No. Discovery and test design start with synthetic or de-identified examples. Later access to production data happens in approved environments under a BAA.
How long does it take to build a healthcare data pipeline?
A diagnostic takes 1 to 2 weeks. A new source typically takes 3 to 6 weeks to build and exercise after a 1 to 2 week design, followed by 1 to 2 weeks of monitoring setup and handover.
What drives the cost of a healthcare pipeline?
Source count and variety, freshness, update semantics, source data quality, the state of any existing pipeline and monitoring expectations. We estimate after the diagnostic.
Talk to our team
Scope a healthcare pipeline review
Tell us what you are planning, what exists today and what needs to change. We’ll review the context and discuss scope, dependencies and the next useful step.
hello@nirmitee.io →Thank you. Your enquiry has been received. Our team will review your requirements.
Explore Related Services