How to Build HIPAA Compliant AI Agents: Architecture and Security Review Guide (2026)

Part of our complete guide to HIPAA checklist for healthcare software developers.

Short answer: To build HIPAA compliant AI agents, get a Business Associate Agreement (BAA) for every service that touches protected health information (PHI), limit what PHI reaches the model, and put separate software between the model and every action it can take. Then record each action in an audit trace and test how the agent fails, because HIPAA judges how the whole system handles PHI and no model is compliant by itself.

Updated October 2026. Vendor BAA terms were checked on vendor pages on 30 October 2026.

This guide is for the CTO, CISO or product lead building an agent that will face a security review. If you want a team to build it with you, see our custom AI agents for healthcare. If you are still choosing a platform, start with which AI agent platforms sign a HIPAA BAA. This guide covers what comes after that choice: building them. It is an engineering guide, so confirm legal conclusions with your counsel.

How do you build HIPAA compliant AI agents? Eight steps

Start by mapping where PHI goes for one workflow, then add the controls in the order below. Building AI agents for healthcare with HIPAA constraints goes faster in this order, because each step produces evidence your security review will ask for.

What does a HIPAA compliant AI agent architecture look like?

A HIPAA compliant AI agent architecture has six layers, and the model is only one of them. The layers that make it defensible are the ones around the model: identity, the PHI boundary, the policy layer and the audit trace.

HIPAA has no AI chapter, so these layers map to existing rules. The Security Rule at 45 CFR 164.312 sets the technical safeguards, and the same layers serve as your HIPAA compliant AI agent infrastructure for healthcare whichever cloud you use. Microsoft notes that HHS has approved no certification program through which a cloud provider could demonstrate HIPAA compliance, and that a BAA does not by itself make your solution compliant.

What counts as PHI once an agent reads it?

Treat any text in the agent's context as PHI if it links a person to their health, care or payment. Once that text reaches an LLM context window, the prompt, the response and anything cached or logged from them need the same protection as the source record.

The 18 HIPAA identifiers and LLM context windows

The Safe Harbor method lists 18 identifier types: names, geographic data smaller than a state, dates (except year) related to an individual, phone numbers, fax numbers, email addresses, Social Security numbers, medical record numbers, health plan beneficiary numbers, account numbers, certificate or license numbers, vehicle identifiers and serial numbers, device identifiers and serial numbers, web URLs, IP addresses, biometric identifiers, full-face photographs, and any other unique identifying number or code. Ages over 89 must also be grouped.

Be careful with the common shortcut that a name alone or a code alone is safe. A diagnosis code with no link to a person is not PHI. A patient list held by a clinic can be PHI with names and no diagnosis, because being on that list says something about the person's care. Decide from context, and check the HIPAA checklist for developers for the wider Privacy and Security Rule duties.

De-identification strategies for AI processing

HIPAA gives two methods at 45 CFR 164.514(b), and data that meets either one is no longer PHI.

• Safe Harbor: remove all 18 identifier types and have no actual knowledge that what remains could identify the person. Both conditions must hold.
• Expert Determination: a qualified expert finds the re-identification risk very small and documents the method and results. This can keep more clinical context, such as partial dates, and often suits AI training data.

Many teams mask identifiers before text reaches a model. This sketch shows the idea only:

# Illustration only: masks four identifier types. Not a Safe Harbor implementation.
import re
PATTERNS = {"SSN": r"\b\d{3}-\d{2}-\d{4}\b", "MRN": r"\bMRN[:\s]*\d{6,10}\b",
            "DATE": r"\b\d{1,2}/\d{1,2}/\d{4}\b", "EMAIL": r"\b[\w.-]+@[\w.-]+\.\w+\b"}

def mask(text):
    token_map, n = {}, 0
    for kind, pattern in PATTERNS.items():
        def swap(match):
            nonlocal n
            n += 1
            token = f"[{kind}_{n}]"
            token_map[token] = match.group()
            return token
        text = re.sub(pattern, swap, text)
    return text, token_map  # token_map stays inside your PHI boundary

The sketch misses street addresses, names, account numbers, device numbers and more, so its output can still contain PHI. For production, test a clinical detection tool such as Microsoft Presidio, Amazon Comprehend Medical or John Snow Labs on your own notes. The token map links tokens back to patients, so keep it inside your PHI boundary with the same access controls as the source record.

LLM PHI de-identification validation

Before you rely on HIPAA Safe Harbor de-identification for LLM input, test it the way an auditor would:

• Run the detector on a labelled set of synthetic notes that covers all 18 identifier types, and count misses per type.
• Document the method you rely on, Safe Harbor or Expert Determination, and who signed it off.
• Re-run the test when note templates, specialties or the detection model change.
• Review a sample of masked text inside your PHI boundary on a set schedule.

Until those checks pass, treat masked text as PHI and send it only to a model covered by your BAA. PHI redaction tooling reduces exposure, and it does not replace the BAA until you can show it meets one of the two methods.

Which AI vendors sign a BAA, and what does it cover?

OpenAI, Anthropic, Google Cloud, AWS and Microsoft all offer a BAA, but each covers only named services and features. Features agents rely on, such as web fetch, file storage or connectors, can sit outside the agreement.

Terms change often, so re-read them before PHI flows and record the date you checked.

The rest of the BAA chain

Any of these needs a BAA, or must receive no PHI, if it touches patient data:

• Agent memory and data stores: vector databases, memory graph platforms for patient relationship data, and any database, including its data placement and backups.
• Observability: tools that log every prompt and agent step, plus error tracking and analytics. Check each vendor for a BAA and for a deployment option inside your environment.
• Integration tools: embedded iPaaS providers, MCP servers and agent orchestration platforms. What decides it is whether PHI passes through their systems.
• Browser infrastructure: hosted browsers for screen-operating agents see every page the agent sees.
• Accounting APIs: an invoice line with a patient name and procedure is PHI. To build HIPAA compliant AI agent integrations with accounting APIs, get a BAA from that vendor or send de-identified references only.
• Annotation tools: labeling projects on PHI-sensitive text need VPC or on-prem deployments, or a BAA.

Apply the same test wherever PHI travels. A digital health startup handling PHI in a managed data pipeline with daily FHIR syncs should check both cloud native and hybrid deployment options, and get estimated run costs in writing, because logging and retention costs change them. A healthcare organization deploying agents for clinical decision support needs the same answers from its memory store. Model monitoring solutions need on-prem or private deployment choices and granular access controls if they log every prompt. Vendor certifications help a reviewer, and none of them replaces a BAA.

Three architecture patterns, and how to choose

Choose Pattern A when PHI cannot leave your environment, Pattern B when the task works without patient identity, and Pattern C when it does not and a BAA covers every vendor in the path. Decide per workflow, because one product often uses more than one.

How to deploy healthcare AI so PHI never leaves

Run an open-weight model on hardware or a private network you control, and block outbound traffic from the inference service. Hospitals asking for the best on premise AI solution will find no single right product: the choice depends on GPU capacity, the staff who will run it, and whether a smaller model passes your evaluation on your own notes. Our on-premise LLM guide goes deeper. To put agents on claims and EHR data without moving PHI out of your environment, keep the model, memory, tools and logs inside it.

# Illustrative network policy: the inference service can reach only labelled internal services
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: llm-no-egress
  namespace: hipaa-ai
spec:
  podSelector:
    matchLabels:
      app: llm-inference
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          zone: hipaa-internal

A network policy only restricts traffic, so you still need encryption, access control, audit logs and patching. Compare monthly GPU cost and staff time against your expected token volume at the API's published price before you commit. For a healthcare startup subject to HIPAA, the safest way to run agentic AI open source tools is the same: self-host them inside your boundary, turn off any telemetry that leaves it, and keep audit logs and RBAC controls outside the agent's reach.

How do you build a HIPAA compliant data layer for LLM agents?

Check the requesting user's and the agent's access to each patient before retrieval, and send the model only the fields the task needs. Filtering after retrieval is too late, because the PHI has already left the store.

• Stop cross-patient retrieval: filter vector search by patient and encounter, and never let the model choose which patient's records to fetch.
• Treat derived data as PHI: embeddings, vector indexes, summaries and agent memory built from PHI need encryption, access control and BAA coverage like the source database.
• Apply minimum necessary: the standard at 45 CFR 164.502(b) means a coding agent does not need a phone number and a scheduling agent does not need lab results. Make the field list code.
• Isolate credentials: each agent gets its own service account, and no two services share a database connection or API key.
• Encrypt with managed keys: use envelope encryption with AWS KMS, Azure Key Vault, Google Cloud KMS or HashiCorp Vault, including for logs, token maps and cached responses.

Encryption under the current Security Rule is an "addressable" specification, which means you must implement it or document why an equivalent alternative is reasonable. Treat it as mandatory in practice, using TLS 1.2 or later in transit and AES-256 at rest as your own standard. Secondary reports say HHS's January 2025 proposal to make encryption required has not been finalized and has moved to a later agenda date; check the Federal Register before relying on either state.

Healthcare autonomous AI agents: controls on what the agent can do

Healthcare autonomous AI agents need limits on actions as well as on data, and those limits must be enforced outside the model. The model proposes an action, and separate software decides whether it is allowed for this patient, this agent and this moment.

Healthcare specific autonomous AI agents can be HIPAA compliant when autonomy is a setting per decision type, granted after evidence. An agent can have a lot of it for low-risk tasks and none for clinical ones.

# Illustrative policy check. It runs outside the model, and any failure stops the action.
def authorize(agent, call, user, audit):
    if call.tool not in agent.allowed_tools or not call.params_valid():
        return audit.deny(call, "tool_or_params")
    if not access.allowed(user, agent, call.patient_id):
        return audit.deny(call, "no_patient_access")
    if call.writes_phi and not patient_match(call, min_identifiers=2):
        return audit.deny(call, "patient_mismatch")
    tier = REVIEW_TIER.get(call.decision_type, "human_decides")  # unknown types get the most review
    if tier != "autonomous":
        return review_queue.add(call, tier)  # a timeout escalates and never approves
    return audit.allow(call)

• Give every agent its own identity. Use short-lived credentials scoped to its workflow. A shared staff login removes the unique user identification HIPAA requires and makes actions untraceable.
• Screen-operating agents need extra care. Every screenshot sent to a model contains PHI, so cover the endpoint with a BAA, crop to the working area, delete screenshots after the step, and confirm the EHR or portal terms allow automation. Every write needs a read-back. See EHR automation without an API.
• Multi-agent systems multiply the paths. In a HIPAA compliant multi-agent AI system architecture, each agent keeps its own identity and scope, PHI passed between agents meets minimum necessary, and one trace ID follows the request.
• Plan for prompt injection. A referral fax, email or web page can carry instructions aimed at the agent. The OWASP Top 10 for Agentic Applications (December 2025) covers goal hijacking, tool misuse and identity abuse. Scoped identities and the policy layer limit what an injected instruction can reach.

Human review: match it to the risk

Set human review per decision type: sampled review for low-risk admin work, approval of each output for moderate risk, and an independent clinical decision for anything that could change care. These tiers are a starting design to agree with your clinical and compliance leads, and they are not regulatory thresholds.

For a medical summarization workflow or any other, fine-grained approval checkpoints belong in the policy layer, with unknown decision types defaulting to Tier 3 and timeouts that escalate instead of approving. Start new decision types one tier higher than you expect, and lower them only with quality data. Clinical decision support can also raise a separate FDA question, covered in its clinical decision support guidance (January 2026), so take that to regulatory counsel.

What should the audit log for an AI agent capture?

Capture who asked, which agent and model version acted, what data it used, what it did, who reviewed it, and what happened. The audit controls standard at 45 CFR 164.312(b) requires you to record and examine activity in systems that contain or use electronic PHI.

{
  "trace_id": "example-trace-0001",
  "timestamp": "2026-10-30T14:23:01Z",
  "agent": {"id": "coding-assistant", "version": "1.4.2", "model": "example-llm-2026-01"},
  "on_behalf_of": "user-1042",
  "patient_ref": "internal-ref-77",
  "context": {"phi_in_context": true, "fields_sent": ["problem_list", "visit_note"], "input_hash": "sha256:example"},
  "action": {"tool": "suggest_code", "output_ref": "store://example/9f86", "policy_result": "allowed"},
  "human_review": {"tier": "approve_each", "reviewer": "user-2210", "status": "approved", "at": "2026-10-30T14:25:33Z"},
  "result": {"read_back_ok": true}
}

This record is illustrative with invented values. The log itself can hold PHI, so protect it like the source record, or store references as above. Use write-once storage such as S3 Object Lock, Azure immutable blob storage or GCS Bucket Lock, and alert on PHI found where it should not be, unusual tool use and sudden changes in override rates.

HIPAA sets no retention period for audit logs themselves. The six-year rule at 45 CFR 164.316(b)(2) covers required documentation, and six years is a sensible default for logs that evidence required reviews; set it in policy. For the design of the trail, see agent audit trails for HIPAA. A business associate must report a breach to the covered entity within 60 days of discovery under 45 CFR 164.410, so your trace also has to tell you which patients an incident touched.

HIPAA compliance and AI governance in US healthcare

HIPAA tells you what to protect, and the NIST AI Risk Management Framework gives you a voluntary way to manage the AI risks around it. State laws then add duties, mostly for data and decisions outside HIPAA's reach.

See NIST's Generative AI Profile for the generative AI detail.

Consent, TPO and sensitive data

Most AI use of PHI for treatment, payment and health care operations needs no new patient authorization under HIPAA, and sensitive data adds limits. Psychotherapy notes generally need specific authorization under 45 CFR 164.508(a)(2), and substance use records fall under 42 CFR Part 2. Training or improving models on patient data can move from operations into research, so take that to your IRB and compliance officer. AI-drafted notes that join the designated record set stay subject to the patient's right of access under 45 CFR 164.524.

State and EU rules to check

In what order do you build it?

Build in four phases: foundation, core controls, validation and production. Your vendors, review boards and integrations set the timeline, so plan by exit criteria.

1. Foundation: run the risk assessment, name the owners, pick the pattern and sign the BAAs before any real PHI flows. Use synthetic patients until then.
2. Core controls: build the data layer, identities, policy layer, review routing and audit trace, then run security penetration testing.
3. Validation: run in shadow mode next to the human workflow, test for bias across demographic groups, rehearse an AI-related PHI breach, and train your users.
4. Production: start at the higher review tier, watch override rates and quality, lower tiers only with evidence, and hold a regular compliance review.

AI agent security review checklist: the packet to bring

An AI agent gets through security review more easily when you bring evidence for each question before it is asked.

Set your bar for unauthorized writes at zero in testing.

Where Nirmitee helps, and where it does not

Nirmitee is a healthcare engineering services company, so we fit if you want a team to build the agent or review its architecture. We can map the PHI flow and BAA chain, build the policy layer and audit trace, and fill in the review packet above. We do not sell a model or a no-code platform, we do not give legal opinions, and no vendor can certify you as HIPAA compliant. Your counsel and compliance officer make the final call.

What you can check: we are ISO 27001:2022 certified and we sign BAAs. Our SMART on FHIR app has run against Epic and Oracle Health developer sandboxes, and our open source headless EHR passed 47 of 47 ONC (g)(10) SMART App Launch Standalone Patient App tests on 27 October 2026 (the SMART group only, not full (g)(10)). When you compare any agency for a HIPAA compliant infrastructure setup or agent build, ask for the same kind of evidence, and ask whether it will sign a BAA.

Bring one agent and its data flow. We will map where PHI goes, mark every gap in the BAA chain and tell you which controls are missing. See our HIPAA-compliant software development work, or book an AI agent security architecture review.