Gartner projects that by 2028, one-third of enterprise software applications will include agentic AI, up from less than 1% in 2024 — and most engineering teams designing agents for healthcare today don't have a canonical reference architecture to work from.
This blog gives you that architecture: the four layers every production healthcare agent we've shipped has needed, the most common design failures, and what to plan for before you write code. For the broader context, see our pillar on AI Agents in Healthcare.
What Is an AI Agent Architecture in Healthcare?
A healthcare AI agent is more than an LLM with a chat interface. It's a system with four components working together: a reasoning core, memory, tools, and orchestration. Each component has healthcare-specific requirements that consumer agent frameworks don't address — HIPAA logging, audit trails, clinical guardrails, FHIR access.
The production patterns we use are what survive HIPAA reviews, on-call rotations, and the inevitable moment a clinical leader asks "why did the agent do that?"
Layer 1: Reasoning Core
The LLM. The part everyone talks about. In practice the model choice is less interesting than the prompt architecture around it. We typically use:
- A strong frontier model (Claude, GPT-4o, or comparable) for the planner.
- A faster, cheaper model for tool-result summarisation and basic classification.
- A specialised clinical or domain-tuned model where there's a clear accuracy gap (e.g., medical entity extraction).
The reasoning core is an architecture decision: which thinking happens where, what's the cost per call, what's the fallback when the model is wrong.
Layer 2: Memory
This is where most agent designs fail. Three kinds of memory exist in a real agent, and conflating them is the most common production bug we see:
- Working memory — scratchpad within a single task. Current goal, intermediate observations, tool call results. Lives for one task lifecycle.
- Episodic memory — what happened in past sessions with this patient/case. "Last time we tried submitting to this payer, they required form X." Structured store keyed on case identifier.
- Semantic memory — long-term knowledge across cases. Payer quirks, policy language, billing rules. Vector store plus structured store working together.
The mistake is putting everything in a single vector database and hoping retrieval figures it out. It won't. Design each type separately.
Layer 3: Tools
Tools are the agent's hands. Every external action — FHIR queries, claim submissions, scheduling, knowledge lookups — is a tool. In healthcare the tool layer is also where compliance lives: audit logging, BAA-protected endpoints, role-based access. See How AI Agents Integrate with EHR Systems for the integration patterns.
Patterns that show up in every production agent:
- Tool granularity matters. "Submit prior auth" is too coarse. "Check eligibility / fetch policy / draft submission / submit" — each as a separate tool — is easier to reason about and audit.
- Tool descriptions are prompts. The description the agent sees when picking a tool is the most important documentation in the codebase.
- Failure modes are tools too. "Escalate to human" is a tool. "Pause and ask for more information" is a tool. Let the agent decide when to use them.
Layer 4: Orchestration
The execution loop. Plan → act → observe → re-plan. This is where you decide what kind of agent you're building:
- Single-agent — one planner doing everything. Simple, easier to debug, fine for narrow workflows.
- Multi-agent with specialisation — separate agents for triage, intake, scheduling, coordinated by a higher-level orchestrator. Better for end-to-end patient journeys. See Multi-Agent AI Architecture for Hospitals.
- Deterministic workflow + agent steps — workflow engine (BPMN, Temporal) defines the skeleton; agents handle reasoning steps. Best fit for high-stakes, audited workflows.
For most healthcare use cases, the third pattern is right. Pure-agent orchestration is too brittle for clinical workflows; pure-workflow can't handle reasoning. The hybrid gets through compliance.
What's NOT in the Architecture (But Should Be)
Things that frequently get treated as afterthoughts:
- Evaluation suite — automated tests with clinical scenarios, run on every model change.
- Observability — every tool call, every reasoning step, every model output captured and queryable.
- Guardrails — pre-action and post-action checks. Especially for anything writing back to clinical systems.
- Human-in-the-loop — designed in, not bolted on.
Real-World Example
The publicly-disclosed Hippocratic AI architecture and the academic literature on agentic AI in clinical settings — including papers from JAMA and Nature Medicine in 2024-2025 — consistently describe these same four layers, with variations in vocabulary. Mayo Clinic's published work on ambient documentation, Cleveland Clinic's AI advisory committee disclosures, and the open-source LangChain/LangGraph reference implementations all converge on this pattern. The architecture isn't proprietary. The discipline to build all four layers from day one is what separates production systems from POCs.
Common Architectural Pitfalls in Production
Three design failures show up most often in healthcare agent reviews:
- Putting everything in a vector store. Vector retrieval is great for semantic memory. It's wrong for working memory (too slow), wrong for episodic memory (recency and identity matter more than similarity), and partially wrong for semantic memory (you also need structured lookups for things like payer codes). Use the right storage for each memory type.
- Tools that are too big. A "submit_prior_auth" tool that does eligibility check + policy fetch + form generation + submission internally looks clean, but the agent can't reason about it. When something goes wrong in step 3, the agent only sees "submit_prior_auth failed." Granular tools give the agent visibility and let it recover.
- No evaluation suite at the orchestration level. Teams test the LLM's reasoning. They don't test the whole agent on representative cases. The agent passes individual reasoning tests and fails the end-to-end scenario. Build orchestration-level evals from day one.
What to Build First
Before writing the agent code, build three things: a small evaluation suite with 20-50 representative cases, a basic observability layer that captures every tool call and reasoning step, and a clearly-scoped first tool. Only then start on the planner. Teams that build these foundations first ship in 4-6 months. Teams that skip them rebuild after 12 months when the first production incident reveals what's missing.
Key Takeaways
- Every production healthcare AI agent has four layers: reasoning core, memory, tools, orchestration.
- Memory is where most designs fail — separate working, episodic, and semantic memory.
- Tools should be granular. Tool descriptions are prompts. "Escalate" and "ask for more info" are tools.
- Use a hybrid workflow + agent orchestration for high-stakes healthcare workflows.
- Evaluation suites, observability, guardrails, and human-in-the-loop are first-class architectural concerns, not afterthoughts.
Call to Action
This blog is one piece of a larger picture. For the full overview, read the pillar guide: What Are AI Agents in Healthcare and How Are They Transforming Care Delivery.
Want to build or evaluate an AI agent for your healthcare product? Get in touch with Nirmitee — we ship FHIR-native, HIPAA-compliant AI agents for US healthtech teams and global hospitals.
