ONNX vs TensorRT vs TFLite: Choosing Your ML Inference Runtime for Healthcare Edge Devices

Your radiology AI model achieves 0.96 AUC-ROC in the research lab — trained on an NVIDIA A100, inference takes 12 milliseconds. Then reality hits: the rural clinic where it needs to run has an Intel NUC with no GPU. The dermatology screening app needs to work offline on a patient's Android phone. The ICU monitoring system runs on a Jetson Nano at the bedside. Same model architecture, three completely different deployment targets — each requiring a different inference runtime.

Choosing the wrong runtime means either unacceptable latency (a 3-second delay on a real-time ECG monitor), unnecessary hardware costs (buying GPU workstations when CPU inference would suffice), or deployment impossibility (trying to run a 400MB model on a device with 64MB of RAM). According to MLCommons benchmarks, the right runtime optimization can deliver 2-6x speedup with less than 2% accuracy loss — the difference between a usable clinical tool and a research prototype.

This guide compares the three dominant ML inference runtimes — ONNX Runtime, TensorRT, and TFLite — through the lens of healthcare deployment requirements. We will cover architecture differences, benchmark real clinical model types, provide export code for each format, and give you a decision framework based on your specific deployment target.

ONNX Runtime: The Cross-Platform Standard

ONNX Runtime (ORT) is Microsoft's inference engine built on the Open Neural Network Exchange format. Its core value proposition is portability: export your model once to ONNX format, and run it on CPUs (Intel, AMD, ARM), GPUs (NVIDIA, AMD), and specialized accelerators (Intel OpenVINO, Qualcomm SNPE) without code changes.

Key Characteristics

• Platform support: Windows, Linux, macOS, Android, iOS, web (WASM)
• Hardware support: CPU, NVIDIA GPU (CUDA), AMD GPU (ROCm), Intel (OpenVINO), ARM (NNAPI), Apple (CoreML)
• Precision: FP32, FP16, INT8 (via quantization toolkit)
• Model format: .onnx (exported from PyTorch, TensorFlow, scikit-learn, XGBoost)
• Typical speedup over native PyTorch: 1.5-3x on CPU, 1.2-2x on GPU

Exporting a Healthcare Model to ONNX

import torch
import torch.onnx
import onnxruntime as ort
import numpy as np
import time

# Example: Chest X-ray classification model (ResNet-50)
model = torch.hub.load("pytorch/vision", "resnet50", pretrained=True)
model.set_mode_to_inference()

# Create dummy input matching your clinical image size
dummy_input = torch.randn(1, 3, 224, 224)

# Export to ONNX
torch.onnx.export(
    model,
    dummy_input,
    "chest_xray_classifier.onnx",
    export_params=True,
    opset_version=17,
    do_constant_folding=True,
    input_names=["image"],
    output_names=["prediction"],
    dynamic_axes={
        "image": {0: "batch_size"},
        "prediction": {0: "batch_size"}
    }
)

print("ONNX export complete")

# Run inference with ONNX Runtime
session = ort.InferenceSession(
    "chest_xray_classifier.onnx",
    providers=["CPUExecutionProvider"]  # or CUDAExecutionProvider
)

# Benchmark
input_data = np.random.randn(1, 3, 224, 224).astype(np.float32)
times = []
for _ in range(100):
    start = time.perf_counter()
    result = session.run(None, {"image": input_data})
    times.append((time.perf_counter() - start) * 1000)

print(f"ONNX Runtime — Mean: {np.mean(times):.1f}ms, "
      f"P95: {np.percentile(times, 95):.1f}ms")

TensorRT: Maximum Performance on NVIDIA Hardware

NVIDIA TensorRT is a high-performance inference optimizer and runtime specifically designed for NVIDIA GPUs. It applies aggressive optimizations — layer fusion, kernel auto-tuning, precision calibration — that deliver 2-6x speedup over generic GPU inference. The trade-off: it only runs on NVIDIA hardware and requires a compilation step that is hardware-specific.

Key Characteristics

• Platform support: Linux (primary), Windows, Jetson (ARM)
• Hardware support: NVIDIA GPUs only (datacenter: A100/H100/L4, workstation: RTX, edge: Jetson)
• Precision: FP32, FP16, INT8, INT4 (with calibration)
• Model format: .engine or .plan (compiled from ONNX)
• Typical speedup over PyTorch: 2-6x on NVIDIA GPUs
• Key limitation: Engine files are GPU-architecture-specific (built for T4 will not run on A100)

Building a TensorRT Engine for Healthcare

import tensorrt as trt
import numpy as np

def build_engine(onnx_path: str, engine_path: str,
                 precision: str = "fp16",
                 max_batch_size: int = 8):
    """Build TensorRT engine from ONNX model."""
    logger = trt.Logger(trt.Logger.WARNING)
    builder = trt.Builder(logger)
    network = builder.create_network(
        1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH)
    )
    parser = trt.OnnxParser(network, logger)

    # Parse ONNX model
    with open(onnx_path, "rb") as f:
        if not parser.parse(f.read()):
            for i in range(parser.num_errors):
                print(f"ONNX parse error: {parser.get_error(i)}")
            return None

    # Configure builder
    config = builder.create_builder_config()
    config.set_memory_pool_limit(
        trt.MemoryPoolType.WORKSPACE, 1 << 30  # 1 GB
    )

    if precision == "fp16":
        config.set_flag(trt.BuilderFlag.FP16)
    elif precision == "int8":
        config.set_flag(trt.BuilderFlag.INT8)
        # INT8 requires calibration data
        config.int8_calibrator = HealthcareCalibrator(
            calibration_data_path="calibration_images/",
            cache_file="calibration.cache"
        )

    # Set dynamic batch size
    profile = builder.create_optimization_profile()
    profile.set_shape("image",
        min=(1, 3, 224, 224),
        opt=(4, 3, 224, 224),
        max=(max_batch_size, 3, 224, 224)
    )
    config.add_optimization_profile(profile)

    # Build engine (this takes minutes)
    engine = builder.build_serialized_network(network, config)

    with open(engine_path, "wb") as f:
        f.write(engine)

    print(f"TensorRT engine saved: {engine_path}")
    return engine_path

# Build FP16 engine for chest X-ray model
build_engine(
    "chest_xray_classifier.onnx",
    "chest_xray_fp16.engine",
    precision="fp16"
)

TFLite: Smallest Footprint for Mobile and Embedded

TensorFlow Lite (TFLite) is designed for on-device inference with the smallest possible memory and binary footprint. Its target is mobile phones, microcontrollers, and IoT devices, where every kilobyte matters. For healthcare, this means skin lesion detection apps, wearable ECG monitors, and offline-capable diagnostic tools in low-connectivity settings.

Key Characteristics

• Platform support: Android, iOS, Linux, microcontrollers (TFLite Micro)
• Hardware support: CPU (ARM, x86), GPU (via delegates: OpenCL, Metal), NPU (via NNAPI, Hexagon DSP), Coral Edge TPU
• Precision: FP32, FP16, INT8, dynamic range quantization
• Model format: .tflite (FlatBuffers, zero-copy deserialization)
• Binary size: ~1MB runtime (vs ~100MB for PyTorch, ~50MB for ONNX Runtime)
• TFLite Micro: Runs on devices with as little as 16KB RAM (Cortex-M microcontrollers)

Converting a PyTorch Model to TFLite

import torch
import tensorflow as tf
import numpy as np

# Option 1: PyTorch -> ONNX -> TF -> TFLite
# (Most reliable path for complex models)
import onnx
from onnx_tf.backend import prepare

# Step 1: Export PyTorch to ONNX (reuse from above)
# Step 2: Convert ONNX to TensorFlow SavedModel
onnx_model = onnx.load("chest_xray_classifier.onnx")
tf_rep = prepare(onnx_model)
tf_rep.export_graph("saved_model/")

# Step 3: Convert SavedModel to TFLite
converter = tf.lite.TFLiteConverter.from_saved_model("saved_model/")

# Apply optimizations
converter.optimizations = [tf.lite.Optimize.DEFAULT]

# For INT8 quantization (smallest model, best for mobile)
def representative_dataset():
    """Provide calibration data for INT8 quantization."""
    for _ in range(100):
        # Use real clinical images for best calibration
        data = np.random.randn(1, 224, 224, 3).astype(np.float32)
        yield [data]

converter.representative_dataset = representative_dataset
converter.target_spec.supported_ops = [
    tf.lite.OpsSet.TFLITE_BUILTINS_INT8
]
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8

tflite_model = converter.convert()

with open("chest_xray_int8.tflite", "wb") as f:
    f.write(tflite_model)

print(f"TFLite model size: {len(tflite_model) / 1024 / 1024:.1f} MB")

Head-to-Head Benchmark: Clinical Model Types

We benchmarked three representative clinical model architectures across all four runtimes. These numbers represent median inference latency for a single sample, measured over 1,000 iterations after a 100-iteration warmup.

*TFLite benchmarks on Pixel 7 (ARM CPU + GPU delegate). All other benchmarks on NVIDIA T4 GPU (TensorRT) or Intel Xeon (PyTorch/ONNX CPU). Clinical NLP models are not well supported by TFLite due to transformer attention layers.

Decision Framework: Choosing the Right Runtime

Comprehensive Benchmark Script

Here is a production-ready benchmarking script that compares runtimes for any healthcare model. Use this to make data-driven decisions for your specific model and hardware. For teams already monitoring model performance in production, this connects to the patterns described in our guide on model monitoring for healthcare AI.

import time
import json
import numpy as np
from dataclasses import dataclass, asdict
from typing import Optional
import os

@dataclass
class BenchmarkResult:
    runtime: str
    precision: str
    mean_ms: float
    median_ms: float
    p95_ms: float
    p99_ms: float
    throughput_qps: float
    model_size_mb: float
    peak_memory_mb: Optional[float] = None

class HealthcareInferenceBenchmark:
    def __init__(self, warmup_iterations: int = 100,
                 benchmark_iterations: int = 1000):
        self.warmup = warmup_iterations
        self.iterations = benchmark_iterations
        self.results = []

    def benchmark_onnx(self, model_path: str,
                        input_shape: tuple,
                        provider: str = "CPUExecutionProvider"
                        ) -> BenchmarkResult:
        import onnxruntime as ort
        session = ort.InferenceSession(model_path,
                                        providers=[provider])
        input_name = session.get_inputs()[0].name
        data = np.random.randn(*input_shape).astype(np.float32)

        # Warmup
        for _ in range(self.warmup):
            session.run(None, {input_name: data})

        # Benchmark
        times = []
        for _ in range(self.iterations):
            start = time.perf_counter()
            session.run(None, {input_name: data})
            times.append((time.perf_counter() - start) * 1000)

        times = np.array(times)
        result = BenchmarkResult(
            runtime="ONNX Runtime",
            precision="FP32",
            mean_ms=round(np.mean(times), 2),
            median_ms=round(np.median(times), 2),
            p95_ms=round(np.percentile(times, 95), 2),
            p99_ms=round(np.percentile(times, 99), 2),
            throughput_qps=round(1000 / np.mean(times), 1),
            model_size_mb=round(
                os.path.getsize(model_path) / 1024 / 1024, 1
            )
        )
        self.results.append(result)
        return result

    def generate_report(self) -> str:
        report = [asdict(r) for r in self.results]
        return json.dumps(report, indent=2)

Accuracy Impact Analysis

The critical question for healthcare: how much accuracy do you lose at each precision level? The answer depends on the model architecture and the clinical task. Our testing across four clinical model types showed the following accuracy impact. For context on acceptable accuracy thresholds, see our guide on model monitoring dashboards for healthcare AI solutions.

PTQ = Post-Training Quantization, QAT = Quantization-Aware Training. QAT consistently preserves more accuracy but requires retraining.

Production Deployment Patterns

The runtime choice affects not just inference speed but your entire deployment architecture. Here are proven patterns from healthcare deployments, connecting to the SRE practices for healthcare that ensure reliability:

Pattern 1: NVIDIA Triton with Mixed Runtimes

# triton_model_repository/
#   chest_xray/
#     config.pbtxt
#     1/model.plan          <- TensorRT for GPU inference
#   ecg_monitor/
#     config.pbtxt
#     1/model.onnx          <- ONNX for CPU fallback

# config.pbtxt for TensorRT model
name: "chest_xray"
platform: "tensorrt_plan"
max_batch_size: 16
input [
  {
    name: "image"
    data_type: TYPE_FP16
    dims: [3, 224, 224]
  }
]
output [
  {
    name: "prediction"
    data_type: TYPE_FP16
    dims: [14]
  }
]
instance_group [
  {
    count: 2
    kind: KIND_GPU
    gpus: [0]
  }
]
dynamic_batching {
  preferred_batch_size: [4, 8]
  max_queue_delay_microseconds: 100
}

Pattern 2: Mobile Health App with TFLite

# Python verification before Android deployment
import tensorflow as tf
import numpy as np

# Load and test the TFLite model
interpreter = tf.lite.Interpreter(
    model_path="skin_lesion_int8.tflite",
    num_threads=4  # Match target device cores
)
interpreter.allocate_tensors()

input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

# Simulate camera input (224x224 RGB image)
test_image = np.random.randint(
    0, 255, (1, 224, 224, 3)
).astype(np.uint8)

interpreter.set_tensor(input_details[0]["index"], test_image)
interpreter.invoke()

output = interpreter.get_tensor(output_details[0]["index"])
print(f"Prediction shape: {output.shape}")
print(f"Top class: {np.argmax(output)}")

Shipping healthcare software that scales requires deep domain expertise. See how our Healthcare Software Product Development practice can accelerate your roadmap. We also offer specialized Healthcare AI Solutions services. Talk to our team to get started.