Healthcare Data Engineering
When finance, quality and operations each report a different number, decisions wait and trust in the data falls. We connect your clinical, claims and operational systems so your leaders get one agreed number for each decision, checked against the source. You start with a fixed-price assessment and see the plan and the cost before you commit to a build.
Book the Healthcare Data Platform AssessmentFor healthtech leaders, data teams, providers and payers.
Teams report different numbers from the same data, and nobody can say which is right.
One written definition and one owner for each number, reconciled to the source.
Your data stays in your accounts. You approve who gets access, and you can remove it.
$3,500 fixed for the assessment. 2 to 3 weeks, credited against the build.
Book the assessment and get a costed plan you can use with any vendor.
The business case
Most data programs stall for business reasons long before the technology fails.
Your clinical, claims and operations systems each hold part of the picture. They were built separately, their records change after the fact, and they define the same measure in different ways. So people rebuild reports by hand and argue about whose number is right.
Our view
Start a healthcare data program with one decision and the metric behind it. Choose the platform third, after the definitions and the source contracts. Buying first tends to give you fast queries over numbers that finance and your clinical leads still dispute.
Two teams can query the same table and report different denial rates, because one counts claims and the other counts service lines. A faster engine does not settle that. A written definition with a named owner does.
Lab results get corrected, claims get reversed and encounters get merged. The design has to say what happens to a number that was already reported, and that decision shapes the whole platform.
Approvals for reporting access and interface changes depend on people outside the project. Plan them first, because no amount of engineering shortens them.
Databricks and Snowflake each support PHI under the right agreement and configuration. Your workload shape and the team who will run it should decide, and many organizations should extend what they already own.
When we would tell you not to build a platform: one source, one report and modest volume. A well-kept model in your existing warehouse will serve you better, and our assessment will say so.
For decision makers
These are the questions that decide whether a data engineering vendor is safe to hire. Here are our answers, and how to test them.
| Risk | How we control it |
|---|---|
| Source access arrives late | Start the requests during the assessment, with a named owner on your side |
| Stakeholders dispute the numbers | Sign off definitions before modeling, and show the reconciliation |
| PHI reaches people it should not | Build the access model first, test it with real roles and de-identify where the use allows |
| Platform costs grow unnoticed | Tie refresh rates to decisions and give each workload a cost owner |
| Knowledge leaves with the vendor | Keep code and runbooks in your repository, and walk your team through a replay |
The first step
Fixed price: $3,500. 2 to 3 weeks, credited against the build.
A fixed-price first step for one decision you need the data to support. You finish with enough evidence to approve, change or stop the build, and the outputs are yours to use with any vendor.
Book the Healthcare Data Platform AssessmentThe assessment is complete when all five outputs are delivered and walked through with your owners. Sending the form starts a conversation, and you pay nothing until you agree the assessment.
How the work runs
Every stage ends with something your team can review and reject. We work on one decision at a time and add sources once the first path holds.
Name the decision, who makes it and what they would do differently with better data.
You review: A one-page decision brief
For each source, record the owner, delivery method, keys, correction behavior and access route.
You review: Source contracts
Write the grain, inclusions, exclusions and owner for each metric before modeling it.
You review: Signed-off metric definitions
Implement ingestion, layers, tests and the semantic model for that single decision.
You review: Code in your repository and test results
Compare counts and control totals with the source, and explain every difference.
You review: A reconciliation report
Set alerts, owners and a replay procedure, then choose the next source from the backlog.
You review: A runbook and a ranked backlog
Related delivery
Public-source life-sciences data moves through collection, name reconciliation and approval before reporting and source-cited AI answers.
Delivery context: a public-data intelligence platform built with Frappe and Power BI, with AI assistants. The case shows source integration, reconciliation and review workflows.
Read the implementation case study →
Cost drivers
The assessment is a fixed $3,500. It takes 2 to 3 weeks and is credited against the build. The build is costed in its plan, and these factors move that cost most.
A Bulk FHIR endpoint, an HL7 feed, an X12 file drop and a database CDC stream each need different tooling and tests.
Years of history to backfill, resource counts and daily change volume shape both engineering and platform usage.
Daily batch is simpler than hourly micro-batch, which is simpler than streaming with ordering guarantees.
Cross-source patient matching and local code mapping to LOINC, SNOMED CT or RxNorm add reviewed rules and test sets.
Safe Harbor is rule-based. Expert Determination adds a qualified statistician and documented risk analysis.
Separate dev, test and production catalogs, access reviews and lineage requirements add setup and evidence work.
For your engineers
Everything your technical team will want to check is here: sources, data layers, definitions and samples of the work.
The short answer
We ingest FHIR Bulk Data exports (NDJSON), HL7 v2 feeds, X12 837 and 835 files and database change streams. We land them in Databricks, Snowflake or your existing cloud warehouse, reconcile every load to its source and publish governed datasets with lineage, in identified or de-identified form. The first step is a fixed-price assessment. A first build then aims for one reconciled data product released in your environment, with a runbook and named owners for the next source.
What we build
Each workstream names the standard, the tool and the evidence your team reviews. Most engagements combine two or three of them around one consumer.
Clients for the FHIR Bulk Data Access $export operation at Patient, Group or system level. We poll the status endpoint, download NDJSON files per resource type and load them incrementally with _since, keeping resource versions and meta.lastUpdated.
Bulk FHIR export from Epic, Oracle Health and athenahealthADT, ORU, ORM and SIU feeds through Mirth Connect or a cloud converter, and X12 837, 835, 277CA and 999 files parsed into claim, line and adjustment tables with CARC and RARC codes preserved for denial analytics.
X12 EDI integration servicesLog-based CDC from EHR, practice-management and billing databases with Debezium, AWS DMS or native connectors, so inserts, updates and deletes arrive in order without nightly full extracts.
CDC for healthcare data warehousesBronze, silver and gold layers on Delta Lake or Snowflake, with patient, encounter, claim and member grain documented, plus OMOP CDM or FHIR-shaped analytics models where research or quality programs need them.
Databricks healthcare implementationHIPAA Safe Harbor removal of the 18 identifier types, or pipelines prepared for an Expert Determination, with date shifting, tokenized linkage keys and a re-identification risk review before data leaves the identified zone.
Safe Harbor vs Expert DeterminationCompleteness, validity, referential and terminology checks against LOINC, SNOMED CT, ICD-10-CM and RxNorm, with column-level lineage in Unity Catalog, OpenLineage or dbt docs so that published fields can be traced to their source where lineage is captured.
Governance, lineage and PHI accessSources handled
A list of connectors tells you little. What matters is how each source delivers changes and what that does to your numbers.
| Source | How it usually arrives | What to plan for |
|---|---|---|
| EHR reporting databases | Scheduled extracts or database replication | Local build differences between sites, and vendor rules on access |
| HL7 v2 feeds (ADT, ORU, ORM, SIU) | Interface engine over MLLP | Site-specific Z segments, resent messages and out-of-order events |
| FHIR Bulk Data | $export to NDJSON files per resource type | Nested resources that fan out into many rows, and which resources your vendor enables |
| X12 837, 835, 277CA, 999 | Clearinghouse file drops | Reversals and adjustments that change totals after the first load |
| Practice management and billing databases | Log-based CDC or snapshots | Hard deletes that snapshots never show |
| Devices, labs and reference files | APIs, flat files and vendor portals | Units, local codes and files that change layout without notice |
Integration boundaries
Clear ownership keeps the first data product on schedule.
Your team or the EHR vendor grants export access, Bulk FHIR client registration, database credentials or SFTP drops. We prepare the technical request and test it.
Writing data back into an EHR or exchanging it with partners is interoperability work, scoped with our HL7, FHIR and X12 integration teams.
Your clinical and quality leads own measure definitions. We implement them as tested code and show every exclusion.
Databricks, Snowflake and cloud subscriptions and their BAAs sit between you and the vendor. We sign a BAA with you for our work with PHI.
Model training, clinical evaluation and serving are a separate stage built on the published datasets.
Compare your options
We recommend the platform after the workload is clear. Most organizations already own one of these; the decision is usually what to add, not what to replace.
| Option | Best fit | Strength for healthcare data | Plan for |
|---|---|---|---|
| Databricks lakehouse | Large FHIR, HL7 and claims volumes, streaming, ML and AI workloads | Delta Lake, Unity Catalog lineage, Spark for nested FHIR JSON, MLflow | Compliance security profile and a Databricks BAA before PHI; cluster and job cost ownership |
| Snowflake warehouse | SQL-first analytics, reporting and partner data sharing | VARIANT for FHIR JSON, Secure Data Sharing, masking and row access policies | Business Critical edition and a Snowflake BAA before PHI; warehouse sizing |
| BigQuery with Google Cloud Healthcare API | Teams on Google Cloud with an operational FHIR store | Streaming export from FHIR stores to BigQuery, built-in de-identify operation | Schema of the exported FHIR tables and query cost controls |
| Operational FHIR store | Applications that read and write FHIR resources in real time | Standard API, SMART on FHIR, Bulk Data export for analytics | It is not an analytics engine; pair it with a warehouse for reporting |
| Existing SQL warehouse or Postgres | A bounded use case with modest volume | No new platform to buy or operate | Headroom for growth and a documented path if volume rises |
Platform subscriptions and cloud usage are contracted directly with the vendor. We scope the engineering and make those dependencies visible in the plan.
Medallion layers
Bronze, silver and gold are useful only when each layer has a rule about what may enter it and who may read it.
| Layer | Its job | Rule to enter | Who reads it |
|---|---|---|---|
| Bronze (raw) | Keep what the source sent, with delivery details, for replay and audit | The delivery is logged | Pipelines only |
| Silver (validated) | One clean, non-aggregated version of each record with types, keys and code checks | Schema, key and terminology checks pass, or the record goes to an exception queue with a reason | Data engineers and approved analysts |
| Gold (business) | Facts and dimensions at a stated grain for one business domain | The output reconciles with the source and the definition is signed off | Analysts and the semantic layer |
Dimension mapping
Most cross-system reporting problems come down to one of these dimensions. Each mapping needs a rule, a test set and a person who settles the exceptions.
| Dimension | The mess in the sources | Conformed result | Who settles exceptions |
|---|---|---|---|
| Patient | Different MRNs per site, merged charts, member IDs in claims | A survivor key with a history of merges and match reasons | Data steward or HIM |
| Provider | NPI, local IDs, group versus rendering provider | NPI-based key with role and effective dates | Provider data owner |
| Payer and plan | Free-text names, several IDs for one plan | A governed payer and plan hierarchy | Revenue cycle |
| Location | Facility, department and bed codes that vary by system | A location hierarchy with effective dates | Operations |
| Clinical codes | Local lab and drug codes | LOINC, SNOMED CT, ICD-10-CM and RxNorm mappings, with the original code kept | Clinical informatics |
| Date | Service, admit, post and paid dates in mixed time zones | A date dimension with each role named | Data engineering |
Semantic models
A metric is finished when its owner has signed these details. The examples show the level of detail we ask for. Your owners set the actual rules.
| Metric | What must be written down | The usual dispute | Typical owner |
|---|---|---|---|
| Initial denial rate | Claim or line grain, first remittance only, which reason codes count | Counting claims in one report and service lines in another | Revenue cycle lead |
| Days in accounts receivable | Which balances, which date starts the clock, how credits are treated | Dates of service versus billing dates | Finance |
| 30-day readmission | Index stays, exclusions, planned returns, and whether the program uses a published measure specification | An internal rate compared against a risk-standardized published rate | Quality lead |
| Open care gaps | Eligible population, measure version, lookback, what closes a gap | Using a different measure year or attribution list | Population health lead |
Refresh frequency
Refresh should follow the decision. Faster is more expensive and harder to run, so each step up needs a person who will act on the fresher number.
| Data | A sensible default | Why | Move faster when |
|---|---|---|---|
| ADT and bed status | Near real time | Staff act within the shift | Rarely needed beyond this |
| Lab results for analytics | Hourly or a few times a day | Trends and cohorts tolerate some lag | An alert depends on the result |
| Claims and remittances | Daily, following file deliveries | The source itself is batch | Almost never |
| Executive and quality reporting | Daily or monthly close | A stable number matters more than a fresh one | Almost never |
| Code sets and reference data | On each published release | Changes are versioned | A mid-cycle correction is issued |
Cleanup problems and fixes
These problems recur across healthcare data. Each fix comes with a check, so you can see it working.
| Problem | Fix | How you can check it |
|---|---|---|
| Duplicate patients across sites | Reviewed match rules with a merge history | Sample review of matches, and a count of open exceptions |
| Mixed lab units | Convert in silver and keep the original value and unit | Value ranges by test code before and after |
| Reversed and adjusted claims | Version by claim control number and apply adjustments in sequence | Paid totals tie to the 835 control totals |
| Resent HL7 messages | Deduplicate on message control ID and event time | Count of duplicates removed per feed |
| Unmapped local codes | A mapping table with an owner and a work queue | Share of records mapped, by source |
| Silent gaps in delivery | A delivery ledger that expects each file or batch | An alert names the missing delivery |
Dashboards people act on
Each dashboard should name who opens it, what they decide and what they do next. If nobody can answer, it should not be built yet. These are examples of how we scope them.
| Who opens it | The question | The action it triggers | Trust signal shown |
|---|---|---|---|
| Revenue cycle lead | Which payer and reason are driving new denials? | Send a work queue to the right team and fix the rule upstream | Last load time and tie-out to the 835 |
| Care manager | Which patients have an open gap and an upcoming visit? | Add the gap to the visit plan | Measure version and attribution date |
| Operations director | Where will capacity run short this week? | Move staff or reschedule | Data freshness and forecast range |
| CFO | What cash should we expect, and from which payers? | Adjust the collections focus | Definition owner and close status |
Forecasting loop
A forecast earns trust through repeated comparison with what really happened. We build the loop along with the model.
Record what a simple method, such as last year or a moving average, would have predicted.
Publish the expected value with its lower and upper bounds.
Name the action and the person who takes it at each level of the range.
Store each forecast, then compare it with actuals when they arrive and against the baseline.
Retrain or retire the model. If it cannot beat the baseline, keep the baseline.
Forecasts for clinical care decisions need clinical review and may raise regulatory questions, so we scope those separately. Operational forecasts are the usual place to begin.
Deliverables
All of it lives in your repository and your accounts.
Versioned ingestion and transformation code with its tests.
Owner, delivery, keys and correction behavior for each source.
Signed-off definitions tied to the semantic model.
Evidence that published numbers tie back to the source.
Roles, masking rules and the results of access tests.
Alerts, owners, replay steps and a ranked backlog.
Samples of the work
You should see what the work looks like before you hire anyone. These samples show the format we hand over. All values here are invented for illustration. They do not come from any patient, client or project.
One row for each field, with the rule and the way to test it.
| Source field | Target column | Rule | Test |
|---|---|---|---|
| HL7 OBX-3 (local code GLU1) | silver.lab_result.loinc_code | Crosswalk to LOINC 2345-7 and keep the local code | No unmapped code without a queue entry |
| HL7 OBX-5, OBX-6 (5.4, mmol/L) | silver.lab_result.value_mg_dl | Convert the unit and keep the original value and unit | Value falls in the agreed range for the test |
| HL7 OBX-11 (C) | silver.lab_result.result_status | C supersedes the earlier version for that order | One current row per order and test |
| X12 835 CLP01 | silver.claim_payment.claim_control_no | Key for sequencing reversals | Paid total ties to the file control total |
What a data owner reads before a release. This run is held because one check failed.
| Check | Expected | This run | Result and action |
|---|---|---|---|
| Lab result files received | 4 of 4 | 4 of 4 | Pass |
| Unmapped local lab codes | 0 without an owner | 37 in the queue, all with an owner | Pass, steward reviews |
| Encounters without a patient | 0 | 12 | Fail: release held, ticket to the source owner |
| Paid total against 835 control total | Equal | Equal | Pass |
Time to first dataset
We do not quote a build duration before we have seen your sources. Four things decide it, and the assessment puts an owner on each.
| What decides it | It moves faster when | It slows down when |
|---|---|---|
| Source access | A person with authority owns the access request from the start | Vendor or security approval starts after the build does |
| Metric sign-off | One named owner can agree the definition | Several committees must agree on it |
| Scope | One decision, one consumer and one or two sources | The first release tries to include every source |
| Source quality | Keys and codes are already reliable | Patient matching or code mapping must be built first |
| Stage | What it holds |
|---|---|
| Sources | EHR reporting databases, HL7 v2 ADT and ORU, FHIR Bulk NDJSON, X12 837 and 835, billing databases through CDC, devices and labs. |
| Intake and delivery ledger | MLLP engine, $export client, SFTP and log-based CDC, with expected deliveries checked against what was received. |
| Bronze, silver, gold | Raw history, validated records, an exception queue, and facts and dimensions. |
| Semantic layer | One definition per metric: measures, dimensions and a named owner. |
| People and models | Dashboards, forecasts, data products and AI datasets. |
| Across every stage | Access roles, masking, lineage, audit and reconciliation. |
| Step | What happens | What it protects |
|---|---|---|
| The lab sends a corrected ORU | Same order, new version. | The correction arrives by the normal route. |
| Bronze keeps both versions | With control ID and received time. | Replay stays possible. |
| Silver validates | Keys, unit and LOINC code. | A failure goes to the exception queue with a reason and an owner. |
| The new version replaces the old | The older version stays as history. | The metric moves once. |
| Reconcile with the source | Counts and control totals. | A mismatch holds the release. |
| The dashboard is released | It shows the last load and the tie-out. | Readers can see how fresh the number is. |
Conformed dimensions let claims and clinical facts be compared by the same patient, provider and date.
| Table | Grain or content |
|---|---|
| Fact: claim line | One line per claim version. Billed, allowed, paid, adjusted. |
| Fact: encounter | One row per encounter. Admit, discharge, disposition. |
| Patient | Survivor key and merge history. |
| Provider | NPI, role and effective dates. |
| Payer and plan | Governed hierarchy. |
| Location | Facility and department. |
| Date | Service, post and paid roles. |
| Code | LOINC, ICD-10-CM and CARC. |
Answer in this order. The first yes is the place to start.
| Question | If yes |
|---|---|
| One bounded use case at modest volume? | Extend your existing warehouse. |
| Already running one platform well? | Stay and add to it. |
| Nested data at volume, streaming or machine learning at the center? | Databricks. |
| SQL-first team, reporting or partner sharing? | Snowflake. |
| None of these? | Run the assessment and test both on your own workload. |
| Role | Decides or does |
|---|---|
| Data owner | Sets priorities and approves who gets access. |
| Clinical and finance owners | Sign off each metric definition and its exclusions. |
| Security lead | Approves PHI handling, BAAs and audit evidence. |
| Engineering team | Builds code, tests, reconciliation and runbooks. |
| Consumers | Accept the dataset and report disputes to the owner. |
Rule: no metric goes live without a named owner.
| Where | What happens there |
|---|---|
| Your environment | Holds your source systems, data platform and PHI, under your platform and cloud BAAs, your roles, your masks and your audit logs. |
| During the assessment | We use schemas, data dictionaries and synthetic samples. We do not use PHI or production credentials. |
| During a build | We sign a BAA with you first. Access uses approved roles, is logged and is the least needed. You grant, review and revoke it. |
| The code | Code and tests use synthetic data and live in your repository. |
Explore the implementation decisions
Compare warehouse and lakehouse approaches against the workload, operating team and data consumers.
Work through source dependencies, platform responsibilities and the path to a usable data product.
Connect source fidelity, governed access and downstream consumption in an implementation architecture.
Primary documentation for the standards and platform features named on this page.
FHIR Bulk Data Access IG ($export and NDJSON) ↗45 CFR 164.514: HIPAA de-identification standard ↗HL7 US Core Implementation Guide ↗X12: healthcare transaction standards ↗HL7 FHIR R4 Provenance resource ↗OpenLineage specification ↗Databricks HIPAA and the compliance security profile ↗Snowflake editions: PHI support on Business Critical ↗Databricks medallion architecture ↗Before we begin
They are the engineering work that moves clinical, claims and operational data from source systems into governed datasets: ingestion from FHIR, HL7 v2, X12 and databases, transformation into documented models, quality and reconciliation checks, de-identification, lineage and the operating runbook. The goal is a dataset your analytics, product and AI teams can use, with reconciliation reports that show where it agrees with the source.
Once paid, you own the custom code we write for you and all of your data. We keep our reusable tools and open-source parts. Your data stays in your accounts, and the custom code, models, metric definitions and runbooks live in your repository.
Not automatically. Databricks suits large nested FHIR and HL7 volumes, streaming and ML. Snowflake suits SQL-first analytics and partner data sharing. A modest use case can run on your existing warehouse. We recommend after reviewing the workload, existing investments, skills and operating cost.
We use the access path each EHR supports for your organization: FHIR Bulk Data $export where it is enabled, HL7 v2 feeds through an interface engine, vendor reporting databases and extracts, or scheduled files. Your organization authorizes the access; we build and test the client, the load and the reconciliation.
A named business owner for each metric, such as a revenue cycle or quality lead. We write the definition down, implement it as tested code and show every exclusion, but your owner decides the rule. FHIR Bulk Data and NDJSON loads feed these metrics, and their _since refreshes are covered in the sources section.
Identified data stays in an access-controlled zone with audit logging. For de-identified datasets we apply HIPAA Safe Harbor, removing the 18 identifier types, or build the pipeline an expert needs for an Expert Determination. We sign a BAA with you for work involving PHI and confirm your platform BAAs are in place.
Interoperability exchanges data between systems, often in real time and often back into the EHR. Data engineering prepares and operates datasets for analytics, reporting and product workloads. Many projects need both, so we define the boundary and connect the two scopes.
Each load is reconciled to the source: record counts, control totals such as billed and paid amounts, referential checks between patients, encounters and claims, and terminology validity. Exceptions go to an owned queue with a reason code, so a published dataset states exactly what it excludes. These checks show agreement with the source. They do not establish clinical correctness, which stays with your clinical owners.
Yes. We publish reproducible, versioned datasets with provenance, access boundaries and quality checks, which is what retrieval, extraction and model training need. Model training, clinical evaluation and serving are scoped as a separate stage on top of that foundation.
The assessment takes 2 to 3 weeks. We do not quote a build schedule before it, because source access and sign-off owners set the pace and differ widely. The costed build plan gives the sequence and its dependencies, once those are known.
The Healthcare Data Platform Assessment is a fixed $3,500. It takes 2 to 3 weeks and is credited against the build. It ends with a costed build plan. The build depends on the number and type of sources, history and volume, freshness, identity and terminology work, the de-identification method and governance requirements. Platform charges are billed by the platform vendor.
Yes. We sign a business associate agreement when our work touches PHI. Nirmitee is ISO 27001:2022 certified, and we start discovery and test design with synthetic or de-identified examples before any production data is involved.
Talk to our team
Tell us what you are planning, what exists today and what needs to change. We’ll review the context and discuss scope, dependencies and the next useful step.
hello@nirmitee.io →Explore Related Services
