Prompt Governance for Clinical AI: Version Control, Guardrails and Audit for Healthcare Prompts

Prompt governance is the set of controls that decides who can change the instructions behind an AI feature, how each change is versioned and reversed, what is recorded when it happens, and which rules no change can remove. In clinical software it answers the question every auditor, customer and clinician eventually asks: which instructions produced this note, who changed them, and can we undo it? If your product cannot answer that in a minute, it does not have prompt governance. It has a text box.

This is the second part of our guide to building a HIPAA compliant AI scribe. Part one covered the full contract, from consent to retention. This part goes deep on the layer that changes most often and is governed least: the prompts. Everything described here is running in an AI documentation assistant we built inside a behavioral health EHR, and each rule is covered by an automated test.

Why prompt governance is a business problem, not just an engineering one

A clinical AI feature earns its keep by being tuned to how a practice writes. A group practice wants a shorter progress note. A clinic that treats adolescents wants family involvement captured differently. A supervisor wants the plan section to always name the next measurable step. Every one of those requests is a prompt change, and customers expect them in days, not in the next release.

That expectation is the opportunity and the risk in one. The product that can safely say yes to clinic-specific tuning without a deployment wins renewals. The product that says yes without controls is one careless edit away from notes that drop risk language, invent details or ignore the format a payer expects. The difference between those two products is governance.

It also decides sales cycles. Health systems and larger groups now run AI governance reviews before they switch a feature on. Buyers of certified health IT have been primed by the ONC HTI-1 decision support rules, which ask developers to describe predictive tools through a defined list of source attributes, and by the NIST AI Risk Management Framework, whose Govern function expects clear roles and accountability for AI behavior. A reviewer who asks "how do you control changes to the AI's instructions?" is asking for exactly the controls below. Having them documented, and demonstrable, shortens that review.

Platform templates and clinic overrides

Every AI workflow in the product, such as a progress note draft, an initial assessment, a treatment plan or the client portal assistant, has a template. Templates come in two tiers.

The edit rule is enforced in one place and checked on every change. A clinic administrator who tries to edit a platform template is refused with a message pointing them to create an override instead. A clinic administrator who tries to edit another clinic's override is refused outright. Anyone who is not an administrator cannot reach the governance actions at all.

The runtime rule is just as short. When a draft is generated, the system looks for active templates for that workflow, takes an active clinic override if one exists, and otherwise uses the platform template, choosing the highest version in either case. If neither exists, generation is refused with a message telling the administrator what to enable, rather than running with no instructions. Because the pipeline often runs from a background job with no signed-in user, the clinic is resolved from the appointment being documented, not from a session.

Guardrails that live below the prompt

The single most important design choice in our prompt governance is that the core safety rules are not in any template. They are a preamble defined in code, one per workflow, and the renderer prepends it to the system prompt on every call. The template an administrator edits is appended after it.

For clinical drafting, that preamble tells the model that it drafts and a licensed clinician decides; that it must use only the source material and context provided; that it must not diagnose or make risk, medical necessity, legal, custody or reporting determinations; that risk language is listed for review, never concluded; and that speaker labels from transcription may be wrong and must never decide what goes in the chart. For the client-facing assistant, the preamble restricts it to portal help grounded in that client's own information.

Two consequences follow. First, administrators can improve wording freely without any path to weakening safety, which is what makes it reasonable to let clinics tune at all. Second, the preamble changes only through a code change, which means code review, tests and a release. That is the right amount of friction for the rules that matter most.

Some rules are too important even for the preamble, because a model can ignore an instruction. Those are enforced after the model: suggested diagnosis codes are checked against the code catalog and unknown codes dropped, and crisis language in the client assistant returns a fixed approved message that the model never writes. For the wider pattern, see our guide to a real-time AI guardrails system.

Prompt version control: append only, including rollback

Good prompt version control follows the same rule as a ledger: you never edit history, you add to it.

• Publish creates a new version with the new text, the author and a change note, and makes it the live version. The previous text stays in history untouched.
• Rollback does not delete or reactivate anything. It creates another new version whose text is a copy of the chosen earlier version, and records which version it copies. Version four can be a copy of version two, and the history shows exactly that.
• Enable and disable switch a template on or off with a reason, and are audited like any other change. Disabling a clinic override sends that clinic back to the platform default immediately.

Why not simply point the live version back at version two? Because then the answer to "what was live on the 14th" depends on reconstructing a sequence of pointer moves. With append-only history, the version number on a generation record identifies one immutable text, forever.

One practical lesson: when you add versioning to a system that already has prompts in production, snapshot what is live as the first history entry before the first new publish. We record it with the note "pre-versioning content", so the history starts from what clinicians actually used, not from the first edit after governance arrived.

What the audit trail records, and what it must not

Every governance action writes an audit event in an AI governance category. HIPAA's audit controls standard asks you to record and examine activity in systems that hold health information, and a prompt change alters what that system does with it.

The audit event never contains the prompt text. It carries a fingerprint: a SHA-256 hash of the system and user prompt, shortened for display. The full text lives in the version history table, which only administrators can read. The split is deliberate. Audit logs are exported, forwarded to security tools and read by more people than the product's own screens, and prompts can contain clinical examples, clinic-specific language or details a clinic considers confidential. The fingerprint proves which text was live, and lets anyone with access to the version table confirm it, without copying that text into every system that receives the audit feed. For how audit trails for AI actions make compliance reviews easier, see our piece on AI agent audit trails. The same thinking applies to operational logs generally, as covered in our guide to HIPAA logging without exposing PHI.

Who can change what

Governance is mostly a permissions problem, and the permissions are narrower than teams expect.

Clinicians are deliberately absent from the edit columns. They are the best source of feedback on what a draft gets wrong, and the worst place to put the edit button, because a change made to fix one note applies to every note that follows. The workflow below turns their feedback into a governed change.

Preview without a vendor call, and without real PHI

Administrators need to see what a change will do before they publish it. Our preview renders the full prompt exactly as the model would receive it, including the pinned preamble, but it does not call the model vendor and it does not use real client data. It fills placeholders with synthetic sample material: a short invented exchange, a sample diagnosis list, sample session details. It also lists every placeholder the template uses and flags any that would render empty.

That last check exists because of a real trade-off. Our renderer treats an unknown placeholder as empty, so that old and new templates can coexist while the context builder evolves. The cost is that a typo in a placeholder name silently removes context from the prompt. Surfacing unfilled placeholders at preview time catches that before it reaches a clinician.

Connecting every note to the prompt that produced it

Governance only pays off if you can trace a specific output back to specific instructions. Every generation writes a usage record with the workflow, the model and vendor, the prompt version, token counts, duration and whether it succeeded, and nothing else: no prompt text, no transcript, no output.

That record is what makes investigation fast. When a clinician reports that drafts started omitting something last Tuesday, the usage records show which prompt version was live for those generations, the version history shows what changed, and the audit trail shows who changed it and why. The same records feed a usage summary by workflow, model, role, day and clinic, which also answers the cost question every product owner eventually gets. A clinic administrator sees only their own clinic.

A change-control workflow clinicians will actually use

The controls above are only useful if the path through them is fast. This is the loop we recommend, and it takes minutes, not a release cycle.

1. Capture the request. A clinician or supervisor reports what the draft gets wrong, with an example the administrator can see in the product.
2. Draft the change with a change note. The note says what is changing and why, in words a reviewer will understand later.
3. Preview. Render the change with sample data, confirm the preamble is present and no placeholder is unfilled.
4. Publish a new version. Scope it to the clinic that asked unless it should become the platform default.
5. Watch the next drafts. Check failures in the usage summary and what clinicians edit on the next few notes.
6. Roll back in one action if needed. The rollback is itself a new version with a reason.

What we would add next

Being honest about the edges is part of governance. Three controls are the natural next steps, and we recommend them to any team building this.

• A replay set before publish. A small library of de-identified or synthetic sessions that every candidate version is run against, with the outputs compared to the live version, turns "looks fine in preview" into evidence. Our guide to testing LLM outputs at scale covers how to build one.
• Two-person approval for platform templates. A platform change reaches every clinic at once, which justifies a second reviewer before publish.
• A readable diff between versions, so reviewers approve a change, not a wall of text.

NIST's Generative AI Profile is a useful checklist for deciding which of these to add first, because it maps generative AI risks to concrete governance actions.

Build it or use a prompt management tool

General-purpose prompt management tools handle storage and versioning well, and for internal tools they are often enough. For a multi-clinic clinical product, the parts that matter most are the parts that have to live inside your application.

The build is smaller than it looks: a template table with a version history table, a resolver, a renderer that prepends the preamble, five commands (create, publish, roll back, enable or disable, preview) and an audit writer. The hard part is not the code. It is deciding the rules above before the first clinic asks for a change.

Prompt governance checklist

1. Every AI workflow has a platform template, and clinics can only override their own.
2. The runtime picks an active clinic override, else the platform default, highest version, resolved from the record being documented.
3. The core safety rules are a preamble pinned in code, not text in any template.
4. Rules a model could ignore are enforced after the model, in code.
5. Publish and rollback both create new versions; nothing is overwritten.
6. Every change writes an audit event with actor, versions, fingerprint and reason, and no prompt text.
7. Preview renders the exact prompt with synthetic data, without a vendor call, and flags unfilled placeholders.
8. Every generation records the prompt version it used.
9. Clinicians give feedback; administrators make changes.
10. You can answer "which instructions produced this note" in under a minute.