Healthcare Data Integration: Refresh Frequency, Cleanup and Reconciliation

Updated October 2026

Healthcare data integration is the work of combining clinical, claims, operational and reference data so that one number means one thing, even though each source refreshes on its own clock. Our view is that speed is the wrong first question. The harder job is proving, on every load, what arrived, what changed and how far behind each source is.

A CIO already knows the standards and the benefits. This guide covers what decides whether the program holds up: refresh frequency by source, where cleanup belongs, how reconciliation gates a release, and who has to own which decision. If you need a team to build it, our healthcare data pipeline services page shows the engineering scope.

Nirmitee is a healthcare software engineering firm that builds data platforms for health systems and health tech companies. If you are planning this work, you can start with our fixed-price Healthcare Data Platform Assessment.

Why does healthcare data integration fail after the connections work?

Healthcare data integration usually fails after go-live, when connected data still disagrees. A live interface proves that bytes moved. It says nothing about whether a total is complete, whether a late correction was applied, or whether two teams define an encounter the same way.

Three failures show up again and again, and each one has a different owner:

• Timing. A dashboard joins this morning's ADT to last month's claims and shows a gap that does not exist.
• Versions. A corrected lab result or a voided claim arrives, and the old row stays in the total.
• Meaning. The same field name carries different rules in two systems, and nothing errors because both values are valid.

Our own guide on why integrated data can be worse than siloed data goes deeper on how those errors multiply once sources are joined. The architecture below is built to catch them before a consumer does.

Reconciled data then has to drive decisions. See how to build a dashboard people act on.

What refresh frequency does each healthcare data source really have?

Each healthcare source has its own refresh behavior, and a late source is often working as designed. Claims cannot arrive before the care is coded and billed, and payer files arrive on the payer's schedule. Plan around those clocks, because no pipeline can make them faster.

Rates vary by site and contract, so treat the middle column as the starting question for each source owner. The chart below follows a single hospital stay through those clocks.

Which clock should a report use?

Store three times on every fact: when it happened, when the source recorded it, and when you received it. With all three, and with old versions retained, you can rerun last month's report as it looked then, and you can show how much of a change came from late data. Without them, a restated number looks like an error and people stop trusting the platform.

Then publish a cutoff rule per measure. A census number can close the same day. A paid-claims number needs a run-out period, and your finance or quality lead should set its length from your own data by measuring how much each past month changed after it first closed. We do not quote a standard figure because it depends on your payers.

How do you choose the right integration method for each source?

Choose the slowest method that still supports the decision the data feeds. Streaming costs more to run, needs on-call cover and is harder to reconcile, so it should earn its place. A bed board may need events. A monthly quality measure does not.

Two details are easy to miss. Change feeds drift, so pair change data capture with a periodic full compare. And vendor routes carry approval time and fees, which belong in the plan from the first week. Our streaming ADT guide covers the case where events do earn their cost.

Where should healthcare data cleanup happen?

Cleanup belongs in a named layer with its own gate, and never in the raw data. Raw stays untouched so you can replay it when a mapping turns out to be wrong, which will happen. Each later layer does one job, which makes a bad number traceable to one step.

Our medallion architecture guide covers the storage pattern. The rules below are the ones we would write into the contract for any layer design:

• Reject with a reason. Move bad rows to quarantine with the failed rule and the source key. Silent drops are the most expensive bug in this work, because totals look fine until someone audits them.
• Map codes, and keep the original. Store the source code next to the LOINC, RxNorm or ICD-10-CM target, plus the mapping version. A mapping table is a governed asset with an owner.
• Normalize units with the value. A glucose in mg/dL and one in mmol/L are both valid numbers. Only the unit tells you which one you are charting.
• Set precedence per field. Saying "the EHR wins" is too coarse. The EHR may win on allergies while registration wins on address and the claim wins on final diagnosis codes. Record which source supplied each value with something like FHIR Provenance or a lineage column.
• Never overwrite a correction. Keep every version and flag the current one, so a voided claim leaves the total and stays in the history.

Patient identity is the cleanup step with the highest stakes, and it gets its own guide: master data and patient matching.

How does healthcare data reconciliation work?

Healthcare data reconciliation compares what a source says it sent with what the platform published, on every load, and holds the release when they differ. It runs at three levels, and a program that stops at the first level will still ship wrong numbers:

The query below is the count and value check for claim lines. It works anywhere a source gives you a control file, and when it does not, ask for one in the feed contract. We ran it against a small synthetic data set: a complete month returns PASS, and a month with a missing row returns HOLD.

-- One row per source, period and measure. Run it after each load, before publish.
WITH source_totals AS (
  SELECT source_system, service_month, COUNT(*) AS rows_n, SUM(paid_amount) AS paid
  FROM source_control_file
  GROUP BY source_system, service_month
),
loaded_totals AS (
  SELECT source_system, service_month, COUNT(*) AS rows_n, SUM(paid_amount) AS paid
  FROM conformed_claim_line
  WHERE is_current_version = 1
  GROUP BY source_system, service_month
)
SELECT s.source_system, s.service_month,
       s.rows_n AS source_rows, COALESCE(l.rows_n, 0) AS loaded_rows,
       s.paid AS source_paid, COALESCE(l.paid, 0) AS loaded_paid,
       CASE WHEN s.rows_n = COALESCE(l.rows_n, 0)
             AND ABS(s.paid - COALESCE(l.paid, 0)) <= 0.01 THEN 'PASS' ELSE 'HOLD' END AS result
FROM source_totals s
LEFT JOIN loaded_totals l
  ON l.source_system = s.source_system AND l.service_month = s.service_month;

Write every result to a ledger table and keep it. Over a few months that ledger becomes your best management report, because it shows which feeds fail, how often and for how long. It is also what an auditor or a skeptical executive will ask to see.

How much difference should you tolerate?

Set tolerance per measure, with the person who answers for that number. Paid amounts in a finance report may need to tie to the cent, while a trend chart for operations can carry a small gap if the gap is shown. A check that a value is well formed does not prove the value is correct, so list what each check catches, and do not let a green light stand in for a sign-off on meaning.

Who should own healthcare data integration decisions?

Business owners should decide what numbers mean, and engineers should decide how they are produced. When engineers settle definitions by default, the platform ends up with a number nobody in the business will defend. The model below names one decider for each recurring decision.

Add one contract per feed: the owner, the expected arrival time, the columns and meanings, how corrections are sent, and how much notice you get before a change. Upstream upgrades are a common cause of breakage, so the notice clause matters. Our anonymized case study on a multi-source data foundation shows governed multi-source data work in another setting.

How to decide who should build your healthcare data integration

Hire the team that can show you its controls before it shows you its tools. The six questions below are the ones we would ask in your seat. The answers describe what we would put in your plan, so hold us to them in writing.

Risks and how we control them

Governance and PHI handling

Each layer gets its own access rule, and the people who build reports see only what their job requires. We sign BAAs, and the design logs access so your compliance team can review it. Discovery and test design start with synthetic or de-identified examples. Access to production data comes later, in approved environments, under a signed BAA. Nirmitee is ISO 27001:2022 certified, which covers how we run our security management. It does not make your platform compliant, so your privacy and security leads still review the design.

Who owns the code and the data

Once paid, you own the custom code we write for you and all of your data. We keep our reusable tools and open-source parts. Our design keeps your data in accounts your organization controls. The custom work includes the data contracts and mappings, the pipeline and quality tests, the replay and recovery procedures, and the freshness monitoring.

What the team looks like

Expect a data architect, data engineers, an integration engineer for HL7 and X12 feeds, an analytics engineer and QA, sized to your sources once the assessment is done. Our team has built 350+ interfaces on Mirth Connect, which is the part of this work most specific to healthcare. Your side supplies a business owner for each published number, a source system owner for each feed, and a privacy or compliance reviewer. A project without those three people on your side will stall, whoever you hire.

What you receive

A build engagement should leave you with running pipelines in your environment, documented grain and definitions, reconciliation reports you can rerun, a runbook, and a handover walkthrough with the people who will operate the platform. Write these into the acceptance criteria.

How to judge a vendor

• Ask them to walk you through a reconciliation report and explain one failed check.
• Ask how they handle a voided claim and a patient merge. Vague answers here predict trouble later.
• Ask what happens when a mapping turns out to be wrong six months in. The right answer involves replay from raw.
• Ask for their definition of done. It should include tied totals and a named acceptance, beyond pipelines running.
• Check any compliance claim. HIPAA has no official certification, so treat that phrase as a warning.

Healthcare Data Platform Assessment: $3,500, fixed

The Healthcare Data Platform Assessment costs a fixed $3,500, takes 2 to 3 weeks and is credited against the build. It gives you a build plan you can take to any vendor, including us. It suits a team with several sources and at least one report that people argue about.

• Source inventory: every source, its owner, access route, refresh behavior and known gaps.
• Target architecture: layers, platform recommendation and the tradeoffs behind it.
• Data model outline: core entities, grain and the definitions that need an owner.
• Risk list: technical, vendor, privacy and operating risks, ranked.
• Costed build plan: phases, acceptance checks and an engineering estimate.

Not included: building pipelines, production PHI access, vendor license or API fees, vendor approvals, legal review and ongoing support. Ask for a Healthcare Data Platform Assessment and bring one decision the data must support.