Fine-Tuning LLMs for Healthcare: When to Fine-Tune on Clinical Notes vs RAG, and How (2026)

Fine-tuning an LLM on clinical notes means training an existing model further on de-identified, clinician-labeled notes so that it reliably produces one kind of output: a coded diagnosis, a note section, a structured extraction or a summary in your house format. It is the right call when the problem is format, labels or terminology on a narrow, repeatable task. It is the wrong call when the problem is missing or changing knowledge, where retrieval (RAG) wins, or when a frontier model with a good prompt already passes your evaluation set, which is more often than teams expect.

The medical research lines up with that split. In Van Veen et al. (Nature Medicine, 2024), adapted LLMs wrote clinical summaries that physicians rated equivalent (45%) or better (36%) than summaries from medical experts, and with enough in-context examples most models beat the best QLoRA fine-tuned model. On narrow note tasks the picture flips: fine-tuning for ICD-10 coding lifted exact-match accuracy from under 1% to 97% (69.2% on real clinical notes).

This guide is for CTOs and engineering leads at US health-tech companies who have to decide, then deliver. It covers the decision, the evidence, a HIPAA-safe data pipeline with code, base model choice, where to train, how to evaluate, what it really costs, and the FDA and ONC rules that apply once a fine-tuned model touches patient care. Updated September 2026.

In brief: when to fine-tune an LLM on clinical notes

If you would rather have us look at your task, data and evaluation plan before you commit to an approach, our healthcare AI development team runs that review as a fixed-scope engagement.

What fine-tuning actually changes (and what it cannot)

Fine-tuning changes how a model behaves, not what it can look up. Anisuzzaman et al. (Mayo Clinic Proceedings: Digital Health, 2025) define it as a process in which "a pretrained model, such as an LLM, is further trained on a custom data set to adapt it for specialized tasks or domains." In practice it teaches output shape, label boundaries, abbreviations and tone. It is a poor way to add facts that change, because every change means retraining and re-validating.

Four techniques matter for clinical work:

• Supervised fine-tuning (SFT). Input and ideal output pairs, such as a discharge summary and the clinician's one-paragraph version. The default for extraction, coding and formatting.
• LoRA and QLoRA. Parameter-efficient versions of SFT. LoRA "can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times." QLoRA goes further: it can fine-tune a 65B model "on a single 48GB GPU while preserving full 16-bit finetuning task performance."
• Preference tuning (DPO). Direct Preference Optimization trains on pairs of better and worse answers. It helps where "good" is a judgment, such as triage or summary quality.
• Continued pretraining. Feeding the model large volumes of unlabeled domain text, as Meditron did with 46,000 clinical practice guidelines and 16.1 million PubMed abstracts. Expensive, and rarely the right choice for a product team.

Which one to use depends on the task. Savage et al. (JMIR, 2025) compared SFT and DPO across clinical tasks and concluded that "SFT alone is sufficient for simple classification tasks with well-defined criteria" (F1 0.98 with Llama 3), while DPO improved triage (F1 0.55 to 0.74), clinical reasoning (7% to 36% accuracy) and summarization, at "approximately 2 to 4 times as many graphics processing unit-hours as SFT."

Full fine-tuning versus LoRA is usually a smaller gap than people assume. In Med42, full-parameter fine-tuning of a 70B model reached 71.9% on USMLE questions against 68.3% for LoRA, which the authors called "remarkably close."

Fine-tuning vs prompting vs RAG: what medical studies show

Across the published medical evidence, large models with good prompts or retrieval win on open-ended questions and summaries, and fine-tuned models win on narrow, repeatable clinical-note tasks. The table below summarizes the evidence.

Where prompting or RAG won

• Clinical summarization. Van Veen et al. tested eight LLMs on radiology reports, patient questions, progress notes and doctor-patient dialogue. GPT-4 with in-context examples was best overall, and in the safety analysis the model made errors on 6%, 2% and 5% of samples across three error categories, against 9%, 4% and 12% for the medical experts.
• Medical question answering. OpenMedLM (Scientific Reports, 2024) used prompting alone on Yi 34B and scored 72.6% on MedQA, above the 65.4% of Meditron 70B, which had been built with costly domain training.
• Fine-tuning vs RAG head to head. Pingua et al. (Bioengineering, 2025) tested Llama 3.1 8B, Gemma 2 9B, Mistral 7B, Qwen 2.5 7B and Phi 3.5 Mini and found "RAG and FT+RAG consistently outperformed FT alone across most models."

Where fine-tuning won

• ICD-10 coding. Hou et al. (npj Health Systems, 2025) trained on 74,260 code-description pairs: exact match went from under 1% to 97%, and reached 69.2% exact and 87.2% category match on real clinical notes. The drop from curated pairs to real notes is the number to plan around.
• Note sectioning. MedSlice (2025) fine-tuned Llama 3.1 8B to split progress notes into sections: F1 0.92, above GPT-4o, and 0.85 on external validation.
• Social needs extraction. Guevara et al. (npj Digital Medicine, 2024): fine-tuned Flan-T5 models identified 93.8% of patients with adverse social determinants of health in their notes, while ICD-10 codes captured 2.0%.
• Small specialized models. Lehman et al. (2023) found "relatively small specialized clinical models substantially outperform all in-context learning approaches" on the clinical tasks they tested.

Where it got complicated

A 2026 JMIR systematic review of 35 clinical studies found hybrid pipelines (fine-tuning plus retrieval) "consistently produced the highest clinical-utility scores for complex tasks," but rated 25 of the 35 studies at high risk of bias. And "LLM is Not All You Need" (2026) found classical models such as logistic regression and LightGBM beat LLMs on most medical categorization tasks, with a lightly LoRA-tuned Gemma 3 performing worst. The practical lesson: published numbers tell you which approach to try first, not which one will win on your notes.

The decision: six questions, in order

Answer these in order and stop at the first one that settles it. Most teams stop at question 2 or 3, which is the cheaper and safer outcome.

1. Do you have a fixed, clinician-labeled evaluation set? If not, build 200 to 500 labeled cases before anything else. Without it there is no way to compare approaches, and no way to prove the model is safe.
2. Does a frontier model with a good prompt pass it? If yes, ship with prompt versioning, output validation and clinician sign-off. Revisit fine-tuning when volume or data-residency changes the economics.
3. Are the failures missing or outdated facts? Then add retrieval from the chart, formulary or guideline and cite the source in the output.
4. Are the failures format, style, labels or terminology? That is what fine-tuning fixes. Plan for 1,000 or more de-identified, labeled examples for one task.
5. Must PHI stay on your own servers, or is per-call cost too high at your volume? Fine-tune a small open model to match the large model on that one task, then run it where the data lives.
6. Can you retrain and re-validate every time the task changes? If not, stay on prompting plus RAG until you can own the model lifecycle.

From our delivery work: a clinical-notes assistant without fine-tuning

We recently built an AI documentation assistant inside a US behavioral-health EHR. It turns a therapy session into draft progress notes (SOAP, DAP, BIRP or general format), initial assessments, treatment plan updates and ICD-10 suggestions, and it is in production. We ran the same six questions and did not fine-tune.

Why not: a frontier model on a HIPAA-eligible cloud (Claude on Amazon Bedrock, a HIPAA-eligible AWS service) already wrote in each note format when the prompt asked for it. The failures that mattered were somewhere else:

• Grounding. Drafts had to use the diagnoses on file, the treatment goals and the prior notes, not general knowledge. We built a context builder over the chart instead of a vector database, and each drafted section points back to its source (a transcript timestamp, a goal, a prior-note line). Unsupported text is flagged.
• Codes. ICD-10 suggestions are limited to the chapters that behavioral health uses and re-matched against the code catalog, so an invented code never reaches the clinician. Nothing enters a note or a claim until a clinician accepts it.
• Governance. Prompts are versioned with publish and rollback, a safety preamble is pinned in code where admins cannot edit it, signed records are locked against AI edits, and usage and cost are reported by feature.

Fine-tuning would have fixed none of those, and it would have added a training-data pipeline for therapy transcripts, which is the most sensitive data the practice holds. We applied the same principle to the claims assistant we built on that EHR's clearinghouse data: a validator refuses any generated sentence that does not cite a fact from the claim record. The clearinghouse side of that engagement is written up in our clearinghouse claims integration case study, and the prompt governance pattern in prompt governance for clinical AI.

Where we would fine-tune on that same platform: a small on-premises model for one extraction task at high volume, such as pulling structured fields out of every note for reporting, once there is a frozen, clinician-labeled test set to prove it matches the large model.

Fine-Tuning an LLM for Clinical Notes: What to Have Ready

Fine-tuning an LLM for clinical notes needs four things before training starts: one defined task, the right to use the notes, protected data and a frozen test set. If any of these is missing, fix it first, because you will not be able to show that the model works or that the data was used properly.

• One task with a pass mark: for example, one extraction or coding job and the score it must reach.
• The right to use the notes: check your contracts and your HIPAA basis, not only your technical access.
• Protected data: 45 CFR 164.514(b) allows two de-identification methods, expert determination and safe harbor. Names and dates can sit anywhere in free text, so check a sample by hand after any automated pass.
• A frozen test set: split by patient and label it with clinicians before training, so results stay honest.

How to fine-tune an LLM on clinical notes: the 8-step pipeline

Training is one step of eight. The gates between the steps are where clinical fine-tuning projects succeed or fail.

Step 1: Define one task and its acceptance threshold

Pick one note type, one output schema and one metric, and get a clinical owner to sign the threshold before any data moves. "Extract medication, dose, route and frequency from discharge summaries with field-level F1 of at least X, and zero invented medications on the test set" is a task. "Make the model better at clinical notes" is not.

Step 2: Secure the right to use the data

Training data is PHI until it is de-identified. You need a business associate agreement with every vendor that touches it, an IRB approval or quality-improvement determination where research rules apply, and a minimum-necessary extract. Put the model's intended use in writing now, because it drives the FDA analysis later.

Step 3: De-identify, then check what you missed

HIPAA gives two routes under 45 CFR 164.514. Safe Harbor removes 18 identifier categories, including names, geographic units smaller than a state, all date elements except year, phone and fax numbers, email addresses, SSNs, medical record numbers, health plan and account numbers, device identifiers, URLs, IP addresses and full-face photos, and requires that you have no actual knowledge the rest could identify someone. Expert Determination has a qualified expert certify that the risk of re-identification "is very small." Many training projects use Expert Determination because Safe Harbor strips the dates that clinical reasoning depends on.

No automated tool is perfect. Microsoft's own Presidio README says "there is no guarantee that Presidio will find all sensitive information." On the i2b2 2014 test set, Norgeot et al. (npj Digital Medicine, 2020) measured recall of 69.84% for the Physionet tool, 87.80% for Scrubber and 99.92% for Philter. That is why we run two passes (rules, then a trained model) and then have a person audit a random sample.

# Two-pass de-identification: rules first, then a transformer NER model.
# Recall matters more than precision here: over-redaction costs a little
# accuracy, a missed identifier becomes PHI inside model weights.
from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine
from transformers import pipeline

analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()
ner = pipeline("token-classification", model="obi/deid_roberta_i2b2",
               aggregation_strategy="simple")

RULE_ENTITIES = ["PERSON", "DATE_TIME", "LOCATION", "PHONE_NUMBER",
                 "EMAIL_ADDRESS", "US_SSN", "IP_ADDRESS", "URL"]

def deidentify(note: str, threshold: float = 0.5) -> str:
    # Pass 1: pattern and rule based detection
    found = analyzer.analyze(text=note, entities=RULE_ENTITIES, language="en")
    text = anonymizer.anonymize(text=note, analyzer_results=found).text

    # Pass 2: contextual PHI the rules miss (a low threshold biases to recall)
    spans = [e for e in ner(text) if e["score"] >= threshold]
    for e in sorted(spans, key=lambda e: e["start"], reverse=True):
        text = text[:e["start"]] + f"[{e['entity_group']}]" + text[e["end"]:]
    return text

# Pass 3 is a person: sample notes from every batch and audit for residual PHI.

De-identify for a second reason too: models can memorize. Carlini et al. (2021) extracted verbatim names, phone numbers and email addresses from GPT-2's training data, including sequences seen only once, and found "larger models are more vulnerable than smaller models." Treat a model fine-tuned on identifiable notes as PHI itself.

Step 4: Split by patient, and freeze the test set first

The most common silent error in clinical fine-tuning is leakage: notes from the same patient in both training and test data, which inflates every metric. Split at the patient level, keep the test set frozen before training starts, and never tune prompts or hyperparameters against it. The FDA's Good Machine Learning Practice principle 4 says it plainly: "Training Data Sets Are Independent of Test Sets."

from sklearn.model_selection import GroupShuffleSplit

def patient_level_split(df, seed=42):
    """df has one row per note with a pseudonymous patient_key column."""
    outer = GroupShuffleSplit(n_splits=1, test_size=0.15, random_state=seed)
    trainval_idx, test_idx = next(outer.split(df, groups=df["patient_key"]))
    trainval, test = df.iloc[trainval_idx], df.iloc[test_idx]

    inner = GroupShuffleSplit(n_splits=1, test_size=0.15, random_state=seed)
    train_idx, val_idx = next(inner.split(trainval, groups=trainval["patient_key"]))
    train, val = trainval.iloc[train_idx], trainval.iloc[val_idx]

    # Hard gate: no patient may appear in more than one split
    assert not set(train.patient_key) & set(test.patient_key)
    assert not set(val.patient_key) & set(test.patient_key)
    return train, val, test

Step 5: Label with clinicians, and measure their agreement

Have a physician, nurse or clinical informaticist write the target output for each training example. Before labeling at scale, have two clinicians label the same 200 to 300 notes independently and measure Cohen's kappa. If two clinicians disagree, the model cannot learn a consistent answer; fix the labeling guideline, not the model.

Step 6: Train with LoRA or QLoRA on a pinned base model

Pin the exact base model version, log a hash of the training data, the configuration and the random seed, so any model in production can be traced back to what made it. A minimal QLoRA setup with Hugging Face PEFT and TRL looks like this:

import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer

BASE = "meta-llama/Llama-3.1-8B-Instruct"   # pin the revision in production

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_compute_dtype=torch.bfloat16)
model = AutoModelForCausalLM.from_pretrained(BASE, quantization_config=bnb,
                                             device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(BASE)

# Each row: {"messages": [{"role": "system", ...}, {"role": "user", "content": note},
#                         {"role": "assistant", "content": clinician_label}]}
data = load_dataset("json", data_files={"train": "train.jsonl",
                                        "validation": "val.jsonl"})

peft_cfg = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                      target_modules="all-linear", task_type="CAUSAL_LM")

args = SFTConfig(output_dir="out/meds-extractor-v1", num_train_epochs=2,
                 per_device_train_batch_size=4, gradient_accumulation_steps=4,
                 learning_rate=2e-4, bf16=True, eval_strategy="epoch",
                 logging_steps=20, seed=42, report_to="none")

SFTTrainer(model=model, args=args, train_dataset=data["train"],
           eval_dataset=data["validation"], peft_config=peft_cfg,
           processing_class=tokenizer).train()

The gate for this step: the fine-tuned model must beat your best prompt-only baseline on the frozen test set. If it does not, you have learned something cheaply; stop here. If the baseline comparison supports fine-tuning, our guide to clinical base model selection helps frame the choice of model to adapt.

Step 7: Evaluate for clinical safety, not just accuracy

Covered in detail in the evaluation section below: task metrics, clinician review of omissions and fabrications, red-team cases, and a memorization probe.

Step 8: Deploy with a human in the loop and a rollback path

Every output is a draft until a clinician accepts it. Log each inference with its input reference, output, model version and the clinician's action; that log is your audit trail and your next training set. Monitor for drift when the source EHR, templates or documentation habits change, and test the rollback to the previous model version before you need it. Our MLOps for healthcare guide covers the monitoring side in depth.

Which LLM to fine-tune for healthcare in 2026

For most clinical-note tasks, start with a general open model of 7B to 9B parameters, and move to a medical base model only if it wins on your test set. Medical models start ahead on exam-style benchmarks, but those benchmarks are multiple-choice questions, not your notes.

Two practical points. First, read the license and the model card's intended-use section before you train; two of the medical models above explicitly warn against production use. Second, if you are a healthcare developer asking "which LLM is already fine-tuned for healthcare", MedGemma is the most current option with published benchmarks, and our healthcare LLM landscape guide compares the medical models in more depth.

Where to fine-tune under HIPAA: managed platforms vs your own GPUs

You can fine-tune on PHI-derived data on a managed cloud service that your provider lists as HIPAA eligible and covers under a signed BAA, or on GPUs you control. What you cannot do is assume a service is covered because it is popular.

A BAA covers the platform, not your design. As Microsoft's own HIPAA page puts it, using the cloud "doesn't automatically impart compliance onto your cloud solutions." Access control, logging, retention and who can download the trained weights are still yours to get right.

How to evaluate a fine-tuned clinical model

Evaluate a fine-tuned clinical model on three things: task accuracy on your frozen test set, clinical safety as judged by clinicians, and behavior on cases designed to break it. Automated text-similarity scores are not enough; Stanford's MedHELM found BERTScore differences under 0.07 between models on 10 of 12 open-ended benchmarks and concluded that "automated NLP metrics may not be adequate."

For scale, Asgari et al. (npj Digital Medicine, 2025) had clinicians annotate 12,999 sentences of LLM-generated clinical notes and found a 1.47% hallucination rate and a 3.45% omission rate, then cut major errors "below previously reported human note-taking rates" by refining prompts and workflows. Their method is a good template for your own review. Red-team cases worth writing for every clinical-note model: documented allergies, pediatric dosing, pregnancy, negation ("denies chest pain"), copied-forward text, and rare conditions your training data under-represents. Our guide to testing LLM outputs at scale shows how to run this as a repeatable pipeline.

What it costs to fine-tune an LLM for healthcare

For a typical LoRA project on clinical notes, GPU time is the smallest line in the budget and clinician time is the largest. The worked example below uses current published GPU prices and clearly stated assumptions.

The assumptions (5 minutes per note, 10 GPU hours) are illustrative; your numbers depend on note length and task difficulty. The shape holds: if clinician time is scarce, a prompt-based design that needs only an evaluation set is usually cheaper, even when per-call inference costs more.

FDA, ONC and Section 1557: the rules once a fine-tuned model touches care

Fine-tuning does not change the regulatory question; intended use does. The same model can be unregulated as a documentation aid and a regulated medical device as a diagnostic recommender.

• Is it a device? The four CDS criteria. FDA's Clinical Decision Support Software guidance (current version issued 29 January 2026) keeps software outside device regulation only if it meets all four criteria, including that it supports "a health care professional" and enables that professional "to independently review the basis for such recommendations" so they do not "rely primarily on any of such recommendations." A model that drafts a note for a clinician to edit is in a very different place from one that flags sepsis risk and triggers action.
• If it is a device: plan for retraining. Fine-tuned models get retrained. FDA's Predetermined Change Control Plan guidance (first issued 4 December 2024, current version 18 August 2025) lets you pre-authorize planned modifications with a description, a modification protocol and an impact assessment, so each retrain does not need a new submission. FDA has authorized over 1,600 AI-enabled devices as of September 2026 and has said it will start identifying devices that use LLMs.
• Good Machine Learning Practice. The 10 GMLP principles from FDA, Health Canada and the MHRA read like a checklist for this guide: representative data, independent test sets, focus on "the performance of the Human-AI Team," and monitoring of deployed models for "re-training risks."
• ONC HTI-1, if you sell certified health IT. The decision support interventions criterion requires 31 source attributes for predictive tools and risk management practices, with updated certified health IT due to customers by 31 December 2024. The HTI-5 proposed rule would remove the AI "model card" requirements; it is not final, so plan for the current rule.
• Section 1557 nondiscrimination. Under 45 CFR 92.210, covered entities must not discriminate "through the use of patient care decision support tools," and since 1 May 2025 must make reasonable efforts to identify and mitigate that risk. Subgroup evaluation is how you show you did.

Keeping a fine-tuned clinical model safe after go-live

A fine-tuned model is a product you now maintain. Plan for these from day one:

• Retrain triggers. New note templates, a new EHR version, new sites or specialties, a code set update (ICD-10-CM changes every October), or a measured drop on the monitoring sample.
• Re-validation on every retrain. The full frozen test set and the clinician review, not a spot check. Under a PCCP, this is the modification protocol.
• Base model end of life. Hosted base models get deprecated; OpenAI's fine-tuning wind-down is a live example. Keep the training data, labels and pipeline so you can move the task to a new base model.
• Feedback as data. Clinician edits to accepted drafts are your best source of new training examples, provided you capture them with the same de-identification and consent rules.

A 90-day plan that ends in a decision, not just a model

1. Days 1 to 15: pick one task, agree the metric and threshold with a clinical owner, sign BAAs, start the data extract.
2. Days 16 to 40: de-identify, split by patient, label the frozen test set (200 to 500 cases) with an agreement check.
3. Days 41 to 55: run the prompt-only and RAG baselines against the test set. If one passes, stop and ship it.
4. Days 56 to 75: only if the baselines fail on format, labels or cost: label the training set and run QLoRA experiments on two base models.
5. Days 76 to 90: clinical safety review, red-team cases, memorization probe, then a limited rollout with clinician sign-off on every output.

The outcome is a documented decision with evidence behind it: prompt, RAG, fine-tune or hybrid, and the numbers that justify it. If you want help getting there, our healthcare AI team designs the evaluation set and runs the baselines with you, and the healthcare AI implementation playbook has the checklists. Book a data and evaluation review and we will tell you which approach your task needs before you spend on training.