ONNX Explained Simply: The “PDF” of Machine Learning Models

You train a model in PyTorch. Your deployment team runs TensorFlow Serving. Your mobile app needs something that runs on a phone, and your edge device only speaks C++. Same model, four different worlds, and none of them understand each other.

This is the problem ONNX was built to solve. Let’s break down what it is, how it works, and the core concepts you need to actually use it, in plain language first, then with a bit of code.

1. The Problem: Every Framework Speaks Its Own Language

Deep learning frameworks (PyTorch, TensorFlow, scikit-learn, and others) each store and execute models in their own way. That creates real friction:

• Training and deployment live in different places. Researchers love PyTorch for flexibility, while production systems often need a lean, fast, dependency-light runtime.
• Hardware is fragmented. NVIDIA GPUs, Intel CPUs, ARM chips, mobile NPUs, and browsers all have their own optimized libraries.
• Lock-in is painful. Rewriting a model in another framework just to deploy it is slow and error-prone.

Without a shared format, you’d need a converter for every framework-to-hardware pair. With a shared format in the middle, you only need one exporter per framework and one runtime per hardware target.

2. The Big Idea: A Common Format for Models

ONNX (Open Neural Network Exchange) is an open standard for representing machine learning models. It was started in 2017 by Microsoft and Facebook, and today it is an open project under the Linux Foundation.

Analogy: think of a PDF. You can write a document in Word, Google Docs, or LaTeX, then export it to PDF, and anyone can open it on any device without caring how it was created. ONNX plays the same role for models: train anywhere, export once, run anywhere.

An .onnx file contains the model's structure and its learned weights, packaged in a framework-neutral way (it's serialized with Protocol Buffers).

3. Core Concepts

3.1 The computation graph

An ONNX model is a graph: a set of operations connected by data flowing between them.

• Nodes are operations (matrix multiply, convolution, ReLU, add…).
• Edges are tensors flowing from one node’s output into another node’s input.

input ──► Conv ──► BatchNorm ──► Relu ──► ... ──► Softmax ──► output

Your model’s forward() function, whatever Python it was written in, becomes this static diagram of math operations.

3.2 Operators and opsets

Every node is an operator from a standard catalog: Conv, MatMul, Relu, Softmax, Gemm, LayerNormalization, and hundreds more. The standard defines exactly what each operator means mathematically, so every runtime implements it the same way.

Operators evolve over time, so ONNX uses opset versions. When you export, you choose an opset (for example, 17). A newer opset gives you more operators and updated behavior, but your runtime must support it.

3.3 Tensors and types

All data in ONNX is a tensor with a defined element type (float32, float16, int64, bool, and so on) and a shape. Shapes can be fixed ([1, 3, 224, 224]) or contain dynamic dimensions (["batch", 3, 224, 224]), which lets the same model handle different batch sizes.

3.4 Inputs, outputs, and initializers

The graph has three kinds of named values:

• Inputs: what you feed the model at inference time.
• Outputs: what the model returns.
• Initializers: the constant data baked into the file, mainly the trained weights and biases.

3.5 Static graph

ONNX stores the graph as it was traced or exported. That is what makes it easy to optimize, but it also means heavily dynamic Python logic (loops or if statements that depend on tensor values) can be tricky to export.

4. ONNX Runtime: The Engine That Runs It

The .onnx file is just a description. To actually run it you need an engine, and the most popular one is ONNX Runtime (ORT), developed by Microsoft.

ORT does two key jobs.

Graph optimization. Before running, ORT rewrites your graph to make it faster without changing results:

• Constant folding: pre-computing parts of the graph that never change.
• Operator fusion: merging several operations (like Conv + BatchNorm + ReLU) into a single faster kernel.
• Redundant node elimination: removing no-ops.

Execution providers. Instead of hard-coding one backend, ORT plugs into hardware-specific accelerators called execution providers:

Execution provider Target CPU Default, runs anywhere CUDA / TensorRT NVIDIA GPUs DirectML Windows GPUs OpenVINO Intel hardware CoreML Apple devices

Get Lumina’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

You write one line to pick the provider, and the same model file runs on different hardware. If a provider doesn’t support an operator, ORT can fall back to the CPU for that part.

5. Hands-On: Export and Run a Model

Step 1: Export from PyTorch

import torch
import torchvision

model = torchvision.models.resnet50(weights="DEFAULT")
model.eval()
dummy_input = torch.randn(1, 3, 224, 224)
torch.onnx.export(
    model,
    dummy_input,
    "resnet50.onnx",
    input_names=["image"],
    output_names=["logits"],
    dynamic_axes={"image": {0: "batch"}, "logits": {0: "batch"}},
    opset_version=17,
)

What happens here: PyTorch runs your model once with the dummy input, records the operations, and writes them as an ONNX graph. The dynamic_axes argument keeps the batch dimension flexible.

Step 2: Validate the file

import onnx

model = onnx.load("resnet50.onnx")
onnx.checker.check_model(model)
print("Model is valid")

Step 3: Run inference with ONNX Runtime

import numpy as np
import onnxruntime as ort

session = ort.InferenceSession(
    "resnet50.onnx",
    providers=["CPUExecutionProvider"],  # or "CUDAExecutionProvider"
)
x = np.random.randn(1, 3, 224, 224).astype(np.float32)
outputs = session.run(None, {"image": x})
print(outputs[0].shape)  # (1, 1000)

No PyTorch is needed at inference time, which is a big win for lean deployments.

Bonus: Build a tiny graph by hand

Seeing a graph built manually makes the concepts click:

import onnx
from onnx import helper, TensorProto

# Two inputs, one output
a = helper.make_tensor_value_info("a", TensorProto.FLOAT, [1])
b = helper.make_tensor_value_info("b", TensorProto.FLOAT, [1])
c = helper.make_tensor_value_info("c", TensorProto.FLOAT, [1])
# One node: c = a + b
add_node = helper.make_node("Add", inputs=["a", "b"], outputs=["c"])
graph = helper.make_graph([add_node], "tiny-graph", [a, b], [c])
model = helper.make_model(graph)
onnx.checker.check_model(model)
onnx.save(model, "add.onnx")

That is the whole idea in miniature: named inputs, nodes using standard operators, named outputs.

6. Handy Tools

• Netron: a viewer that draws your .onnx graph so you can inspect every layer, shape, and weight. Great for debugging exports.
• ONNX Runtime quantization tools: convert weights from float32 to int8 or float16 to shrink the model and speed it up, with a small accuracy trade-off.
• Converters: tf2onnx for TensorFlow, skl2onnx for scikit-learn, and similar packages for other libraries.

7. Common Pitfalls

• Unsupported operators. Some custom or very new PyTorch ops have no ONNX equivalent, so the export fails or needs a workaround.
• Dynamic control flow. Data-dependent Python branching may get “frozen” during tracing, so always test the exported model against the original.
• Opset mismatches. A model exported with a high opset may not run on an older runtime.
• Numerical differences. Results should match closely but not always bit-for-bit. Compare outputs with a tolerance (for example np.allclose).
• Not always the best path. For very large language models, specialized inference stacks are often used instead, although ONNX Runtime has tooling for generative models too.

A good habit: after every export, run the same input through both the original model and the ONNX version and compare the outputs.

When Should You Use ONNX?

Good fit:

• You want to decouple training from deployment.
• You need to run on varied hardware (CPU, GPU, mobile, browser).
• You want faster, lighter inference without shipping a whole training framework.

Maybe not:

• Your model relies on exotic custom operators.
• You are staying entirely inside one ecosystem that already serves your needs well.

Key Takeaways

• ONNX is a universal file format for ML models: a graph of standard operators plus the trained weights.
• Core pieces: computation graph, operators and opsets, tensors, inputs/outputs, and initializers.
• ONNX Runtime executes the graph, optimizes it, and targets different hardware through execution providers.
• Workflow: train in your favorite framework, export to .onnx, validate, then deploy anywhere ORT runs.

ONNX won’t make your model smarter, but it makes your model portable, and in production, portability is often the difference between a great experiment and a shipped product.

References

• ONNX project: onnx.ai
• ONNX Runtime documentation: onnxruntime.ai
• PyTorch ONNX export guide: pytorch.org/docs/stable/onnx.html
• Netron model viewer: netron.app