Skip to main content

M07 Overview | Backend Deployment Overview: After the Q/DQ Graph

Open In Colab

M01 to M06 all end at the Q/DQ graph produced by convert_pt2e. It marks “where to quantize and with which qparams”, but the operations themselves are still fp32 conv and linear, and it does not get faster (M02-3). To get faster, something has to replace the quantized operations on the graph with real int8 kernels. M07 to M10 answer exactly this: who does the replacing, at which step, and what to do about what cannot be replaced.

This overview has two halves:

The questions M07 to M10 answer

Question One-sentence answer Chapter
Can a Q/DQ graph become integer arithmetic inside PyTorch? An operation only becomes integer if a pattern matches; otherwise fp32 is kept M07-2
What should you know before comparing the numbers of two implementations? Just swapping the kernel makes fp32 differ by 1e-5 to 1e-4; that is the floor M07-1
What happens when you hand it to ExecuTorch + XNNPACK? The subgraphs are cut at export time and stored as .pte; at run time XNNPACK’s int8 kernels are used M08
What happens when you hand it to ONNX Runtime? Export a QDQ ONNX; ONNX Runtime only replaces the units with int8 ops when it builds the session M09
How should a Quantizer be written when the hardware has more restrictions? Write the hardware constraints as annotations, then check the Q/DQ graph and fall back unsupported ops to the CPU M10

An example first: a unit on the Q/DQ graph

The smallest conv + relu after PTQ (XNNPACKQuantizer, per-channel weight); this is the Q/DQ graph after convert:

Look at the middle part: the input is first dequantized to fp32, goes through the fp32 conv2d and relu, and is then quantized back to int8. Such a “dq → op → q” segment is called a unit:

int8 ─ dq ─→ fp32 ─ conv2d, relu ─→ fp32 ─ q ─→ int8

The weight side is the same: the int8 constant goes through dequantize_per_channel back to fp32 before entering conv2d. The unit’s semantics is int8 in, int8 out, but the operations in the middle are still fp32. To get faster, something has to recognize this unit and replace it with one int8 conv-relu kernel.

Every backend in M08 to M10 consumes a Q/DQ graph made of such units. First we look at who this graph gets handed to (first half), then at what PyTorch itself can turn the units into (second half).

From the Q/DQ graph to the hardware: three layers

ExecuTorch, ONNX Runtime, XNNPACK, and oneDNN are often lumped together as “backends”, but they are not on the same layer. Going down from the Q/DQ graph, the path is:

Q/DQ graph            the endpoint of PT2E (M01 to M06)
  → deployment file   .pte / .onnx / OpenVINO IR
  → inference framework   ExecuTorch / ONNX Runtime / OpenVINO
  → hardware extension unit   backend / Execution Provider / device plugin
  → kernel library    XNNPACK / oneDNN / MLAS ...
  → hardware          CPU / GPU / NPU

The three middle layers are defined as follows:

  • Inference framework (often also called a runtime): reads the deployment file, decides who gets each piece of the graph, manages memory, and schedules execution. It does not necessarily compute conv itself.
  • Hardware extension unit: the socket through which an inference framework connects to some hardware or library. The three frameworks name it differently: ExecuTorch calls it a backend (the subgraph handed out is a delegate), ONNX Runtime calls it an Execution Provider (EP), and OpenVINO calls it a device plugin.
  • Kernel library: the code that actually computes conv and matmul. It does not read model files, so an inference framework must call it.

Inference frameworks

ExecuTorch ONNX Runtime (ORT) OpenVINO (OV)
Led by Meta / PyTorch Microsoft Intel
Deployment file .pte .onnx OpenVINO IR (.xml graph + .bin weights)
Hardware extension unit backend Execution Provider device plugin
When it decides who gets what At export time; the partitioner cuts the graph first At session creation, following the EP order At model compile time, following the specified device
Main target Phones, edge devices Cross-platform, cross-framework Intel CPU, GPU, NPU

The row “When it decides who gets what” is the one most worth remembering: it determines at which step the int8 kernel appears, and M08 and M09 each look at it once.

Kernel libraries

Kernel library Developer Main execution target Who calls it in this project
XNNPACK Google CPUs on arm / x86 / WebAssembly ExecuTorch’s XNNPACK backend (M08)
oneDNN (formerly MKL-DNN) Intel Mainly x86 CPUs PyTorch’s mkldnn, torchao’s x86 lowering (M07-2)
MLAS Microsoft CPU ORT’s default CPU EP (M09)
NNPACK Open-source project CPU (fp32) PyTorch eager’s conv fast path (M07-1)

On arm you also commonly see Arm’s own ACL (Arm Compute Library) and KleidiAI. Tried on openvino 2026.4.1, macOS arm64: OpenVINO’s CPU plugin uses ACL for conv and KleidiAI for some MatMuls.

One Library, Several Frameworks

A hardware extension unit is only an adapter, and the same kernel library can be plugged in under several frameworks: XNNPACK is an ExecuTorch backend and also an ORT EP. But whether an EP is usable depends on whether the ORT build you installed has it compiled in: the pip onnxruntime, onnxruntime-gpu, and an ORT you built from source each have different EPs compiled in. ORT provides two functions to answer this:

  • ort.get_all_providers(): the names of all EPs this version of ORT knows about, regardless of your machine.
  • ort.get_available_providers(): the EPs actually compiled into the build you installed and usable, in ORT’s default priority order.

When a build has several EPs, which one is used is specified at session creation by ort.InferenceSession(..., providers=[...]), with earlier entries in the list taking priority. M09 always specifies providers=["CPUExecutionProvider"], so whether or not your build has the XNNPACK EP, the int8 kernels M09 sees are always the CPU EP’s MLAS, a different implementation from M08’s XNNPACK.

On the ExecuTorch side, a backend is a subpackage under executorch.backends. List them to see (requires uv sync --group executorch):

import importlib.util
from pathlib import Path

if importlib.util.find_spec("executorch") is None:
    print("executorch is not installed (uv sync --group executorch)")
else:
    import executorch.backends

    root = Path(executorch.backends.__path__[0])
    backends = sorted(p.name for p in root.iterdir()
                      if p.is_dir() and p.name not in {"test", "transforms", "example", "__pycache__"})
    print(f"{len(backends)} folders under executorch.backends:")
    print(backends)
    assert "xnnpack" in backends
16 folders under executorch.backends:
['aoti', 'apple', 'arm', 'cadence', 'cortex_m', 'cuda', 'mediatek', 'mlx', 'native', 'nxp', 'openvino', 'qualcomm', 'samsung', 'vulkan', 'webgpu', 'xnnpack']

Each folder is one backend (or a group of backends): xnnpack is the CPU backend M08 uses, Core ML lives under apple, qualcomm is QNN, openvino is OpenVINO … XNNPACK is just one of them.

Behind the adapter, there is not always a kernel library

Several names are both an ExecuTorch backend and an ORT EP: OpenVINO, Core ML, QNN. They are themselves another inference framework or runtime, and the subgraph handed to them is dispatched once more:

Name What it is Where it plugs in After handing over
XNNPACK Kernel library ExecuTorch backend; ORT EP Runs CPU kernels directly
OpenVINO Intel’s inference framework ExecuTorch backend; ORT EP OpenVINO then picks the device plugin and kernel
Core ML Apple’s inference framework ExecuTorch backend; ORT EP Core ML then picks CPU, GPU, or Neural Engine
QNN Qualcomm’s runtime ExecuTorch backend; ORT EP QNN then picks a Qualcomm compute unit

Who calls whom

This draws the relationship between inference frameworks and kernel libraries together (how the deployment files come about is left to the next section). Light red is an inference framework, yellow is a kernel library. Solid lines are relationships that could be verified in the versions used when writing this (package directories, EP names, source code, or a trial); dashed lines come only from documentation: OpenVINO using oneDNN on x86 is what the documentation says, and it was not tried on x86.

Three routes: starting from the Q/DQ graph

PT2E does not produce these three deployment files directly. The Q/DQ graph goes through each framework’s own bridging step to become a file the framework can read. Blue is the Q/DQ graph, green is a deployment file or model IR, light red is an inference framework; the dashed line was only tried with an fp32 model (ONNX → ov.Model):

The line from the Q/DQ graph straight to oneDNN does not go through an inference framework: torchao’s x86 lowering rewrites the Q/DQ graph directly into an fx graph that calls oneDNN (the M07 chapter proper, in the second half). As a table:

Route A: ExecuTorch Route B: ONNX Runtime Route C: OpenVINO
Starting point on the PyTorch side The Q/DQ graph from convert_pt2e The Q/DQ graph from convert_pt2e The graph from convert_pt2e(..., fold_quantize=False)
Bridging step torch.export again, then to_edge_transform_and_lower torch.onnx.export(..., dynamo=True) ov.convert_model
Deployment file .pte QDQ ONNX (.onnx) ov.Model, which can be saved as OpenVINO IR
Who recognizes dq → op → q, and when XNNPACK’s partitioner, at export time ORT’s QDQ node fusion, at session creation Not checked (q/dq first become OV’s FakeQuantize)
In this project M08 M09 Only a conversion trial was done (see below)

Route A fixes the kernels at export time: what is stored in the .pte is exactly the subgraph XNNPACK will execute. Route B’s .onnx is just the Q/DQ graph written in another format; turning it into int8 ops only happens when ORT builds the session, and whose kernels are used depends on the EP.

Route C: OpenVINO first converts to ov.Model

No later chapter covers OpenVINO, so here are a few things tried on openvino 2026.4.1, macOS arm64 (openvino is not a dependency of this project):

  • OpenVINO does not consume PyTorch models directly: core.compile_model(nn.Module, "CPU") fails, and you have to convert with ov.convert_model first.
  • The Q/DQ graph must be produced with convert_pt2e(..., fold_quantize=False) to convert: the default approach stores weights as int8 constants, which gives a type error during conversion.

The result of conversion and the result of compilation are two different things:

Term What it is Related to the device?
ov.Model The result of ov.convert_model; the in-memory form of the OpenVINO IR No; can be saved and moved to another machine
OpenVINO IR The .xml (graph) + .bin (weights) saved by ov.save_model No
CompiledModel The result of core.compile_model(ov.Model, device) compiled for one device plugin Yes (for example "CPU")

OpenVINO can also be embedded into the other two routes: as ORT’s OpenVINO EP, or as ExecuTorch’s openvino backend (the table in “Behind the adapter, there is not always a kernel library”).

Before Quantizing, Choose the Backend

prepare_pt2e(model, quantizer) does not choose a backend for you. The Quantizer decides which ops to quantize, which qspec to use, and which tensors share qparams (M03), and these rules are designed around some backend’s kernels:

Choose the deployment backend
  → build that backend's Quantizer      decides quantization locations and qspec
  → prepare_pt2e, calibrate, convert    get the Q/DQ graph
  → that backend's lowering             recognizes dq → op → q, swaps in an int8 kernel
Target backend Matching Quantizer In this project
XNNPACK XNNPACKQuantizer (there are two with the same name; the difference is in M08-1) M01 to M08
x86 (oneDNN) X86InductorQuantizer M03-7
OpenVINO OpenVINOQuantizer (shipped with ExecuTorch’s openvino backend, built on Intel’s NNCF) Not used
A fictional VPU The hand-written MyVPUQuantizer M10

These Quantizers all inherit from the same torchao Quantizer base class (M03); they differ in the annotation rules.

The Q/DQ graph format is backend-neutral, but the quantization locations and qspec on the graph are not. When the Quantizer and the lowering do not match, some units cannot be recognized and stay on the fallback path: the conv in the reference representation in the second half is one example (one of the mismatched conditions: the pattern needs per-tensor weight, while XNNPACKQuantizer gives per-channel). M10 stands on the backend side the other way around, and checks which constraints of the fictional VPU a Q/DQ graph violates.

The M07 chapter proper: what else a Q/DQ graph can become inside PyTorch

All three routes above leave PyTorch. Before leaving, PyTorch and torchao themselves have three ways of handling a Q/DQ graph, each answering “what happens to this unit next”.

Three terms: not a pick-one-of-three on the same layer

Term What it is Provided by Purpose
reference representation Another output format of convert_pt2e torchao Writes the integer semantics of quantization with ordinary ATen arithmetic
lowering Replacing patterns on the graph with some backend’s ops Each backend implements its own Lets the backend run with its own kernels
Inductor The default compiler of torch.compile PyTorch Compiles the whole graph and generates kernels

The reference representation is a “spec” and does not aim to be fast; lowering is the action of “swapping ops”; Inductor is a compiler, and the x86 lowering happens to borrow its pattern mechanism (see “How the three relate” below).

Reference representation: writing Q/DQ as explicit arithmetic

convert_pt2e(..., use_reference_representation=True) replaces every quantize_per_tensor / dequantize_per_tensor node on the graph with a series of ordinary ATen arithmetic nodes. Use the same conv_relu and compare the segment feeding conv2d’s input. The first figure is the Q/DQ graph, where x goes through one q and one dq into conv2d:

After switching to the reference representation, the same segment becomes 11 arithmetic nodes. It is drawn in two figures, with the constant parameters (scale, zp, dtype) under each node:

conv2d itself is unchanged: it does not match torchao’s pattern (the table below) and still consumes fp32. Only q and dq are replaced. These two series of arithmetic are the two formulas below. With scale \(s\), zero point \(z\), and integer range \([q_{min}, q_{max}]\):

\[ q = \operatorname{clamp}\big(\operatorname{round}(x / s) + z,\ q_{min},\ q_{max}\big) \tag{1} \]

\[ \hat{x} = (q - z)\cdot s \tag{2} \]

Here \(x/s\), round, and \(\times s\) are still floating-point operations: this step is “writing the formula out”, not switching to an int8 kernel.

How the operation in the middle of a unit is handled depends on whether the whole unit matches a pattern torchao has written:

Case The unit becomes Example on torchao 0.18 (M07-2 B)
No match int8 → dq arithmetic → fp32 operation → q arithmetic → int8 The conv in the conv_relu above
Match int8 → int32 accumulate / arithmetic → requantize → int8 add + relu, linear with per-tensor weight

PyTorch officially positions it as a backend-agnostic reference format (PT2E docs, RFC-0019):

  • Good for: spelling out the semantics of integer accumulation, the bias scale, requantize, and clamp; serving as a common reference when backend or compiler authors implement lowering; running and checking the integer quantization logic without target hardware.
  • Not good for: measuring final latency (what you measure is only the speed of the reference arithmetic); predicting the actual performance of oneDNN, XNNPACK, or an NPU; guaranteeing rounding / overflow behavior identical to specific hardware.

Lowering: the int8 op is swapped in only if the pattern matches

Lowering edits the graph at compile time; it does not ask “is there a kernel?” at run time:

decompose        rearrange the graph into forms the patterns recognize (conv2d → convolution, etc.)
pattern match    run the patterns the backend registered in advance, e.g. dq → conv → q
replace          matched ones become the backend's op (x86: onednn.qconv_pointwise)
leave the rest   anything not registered or not matched stays on the graph as is

Each backend has its own lowering:

Backend Lowering entry point Units that do not match Which chapter
x86 (oneDNN) lower_pt2e_quantized_to_x86 Left as is on the fx graph, run in fp32 M07-2 C
XNNPACK to_edge_transform_and_lower(..., XnnpackPartitioner) Left outside the delegate, run by ExecuTorch’s portable kernels M08
ONNX Runtime Export a QDQ ONNX; ORT does QDQ fusion when building the session Left in fp32, with one int8 ↔︎ fp32 conversion before and after M09
A fictional VPU The hand-written partitioner Fall back to the CPU M10

Inductor: the compiler of torch.compile

Inductor is the default backend of torch.compile. It does decompose, pattern matching, constant folding, fusion of elementwise operations, and layout and memory planning over the whole graph, and finally generates C++ kernels on the CPU; operations like conv and matmul call external libraries (oneDNN, BLAS, NNPACK, etc.). When it meets a Q/DQ graph there are two outcomes:

Case What Inductor does
There is an int8 pattern, and freezing is on dq → conv → q is replaced by oneDNN’s int8 conv (same as lowering)
There is no int8 pattern Elementwise operations such as dq, q, and relu are fused into the kernel it generates; conv still calls the fp32 kernel

How the three relate

The x86 int8 patterns are not in PyTorch itself but in torchao (from reading the source of torchao 0.18 and torch 2.14):

Step Where What it does
The pattern itself torchao’s x86.py (inductor_passes/) Replaces dq → conv → q with oneDNN’s int8 op (for example qconv_pointwise)
Registration When x86_inductor_quantizer is imported Registered only when mkldnn is available, into Inductor’s freezing patterns
Who runs it (1) torchao’s lower_pt2e_quantized_to_x86 Calls the freezing pass directly and returns the edited fx graph
Who runs it (2) torch.compile Runs only when Inductor’s freezing setting is on and under no_grad; freezing is off by default (torch’s compile_fx.py)

Rules Before Comparing Backends

After M07-1, every chapter compares some implementation’s result against the Q/DQ graph or the fp32 model. Four things need to be settled before comparing.

The fp32 numeric floor

When comparing two implementations (Q/DQ graph vs. reference, eager vs. lowering, PyTorch vs. ONNX Runtime), the difference is not all caused by quantization. For a completely unquantized fp32 model, as soon as the CPU switches its conv implementation (for example, a different batch size goes through NNPACK instead), the output differs by 1e-5 to 1e-4. This magnitude is the numeric floor: a gap smaller than it is not a fault of quantization or lowering. M07-1 measures it and reproduces it on any machine with the torch.backends switches.

Numerical Differences from the Q/DQ Graph

The Q/DQ graph simulates quantization in fp32, while the backend’s integer kernels really compute with integers, and rounding happens in different places. Another known source is the bias: the bias in the Q/DQ graph is fp32, XNNPACK quantizes it to int32 during lowering with scale = \(s_x \cdot s_w\) (executorch 1.5.0’s quant_params.py, from_bias), and ORT’s ORT_ENABLE_BASIC level does the same thing (M09-2).

So when comparing backends, look only at the final output:

Metric What it answers
SQNR, accuracy against fp32 Whether accuracy got worse
Fraction of outputs bit-identical How close it is to the Q/DQ graph
argmax agreement rate Whether the classification result changed
Max difference in output LSBs Whether the difference is just rounding

The speed baseline: fp32 in the same runtime

The Q/DQ graph itself is slower than eager (it adds fp32 q and dq operations); an fp32 model moved to another runtime can also be much faster. So the gain of int8 should be compared with the fp32 model in the same runtime (M08-4). None of M07’s three treatments produces an int8 kernel, so this chapter does not measure latency.

Deploying to a device often needs static shapes

This project’s export_module() (a wrapper in common/pt2e.py, not a torch API) enables dynamic batch by default, because calibration feeds different batch sizes. ExecuTorch’s .pte lays out memory according to the shapes at export time, so M08 uses export_module(..., dynamic_batch=False); ONNX export needs dynamic_shapes specified again (M09-1); M10’s fictional VPU only takes static shapes, and validate() rejects a dynamic batch.

Platform notes

The outputs stored in this overview and in this chapter’s notebooks were produced on macOS arm64, torch 2.14.0, torchao 0.18, executorch 1.5.0, onnxruntime 1.30.0, without mkldnn: the x86 int8 patterns are not registered, and none of M07’s three treatments turned conv into an int8 kernel. When you run it on your machine, ExecuTorch’s backend list and the notes under the tables are printed according to the measurements. The actual results of lowering and torch.compile on x86 (with mkldnn) were not verified here.

Summary and map

  • Backends have three layers: the inference framework reads the deployment file and dispatches subgraphs; the hardware extension unit is an adapter; the kernel library does the computing. The same name may appear on several layers.
  • All three routes start from the same Q/DQ graph; they differ in the bridging step and in “who, at which step” replaces dq → op → q with an int8 kernel.
  • The Quantizer must match the backend: the quantization locations and qspec are designed around the backend’s kernels, and when they do not match, units stay on the fallback path.
  • Inside PyTorch, the reference representation writes out the math of quantization, and only units that match a pattern become integer; lowering replaces matched patterns with the backend’s int8 ops; Inductor compiles the whole graph and, with freezing on and a matching pattern, also swaps in int8 kernels.
  • Before comparing backends, know the fp32 floor, compare only the final outputs, and compare speed against fp32 in the same runtime.

The notebooks of this chapter:

notebook Question it answers
M07-1 The fp32 numeric floor How much fp32 differs when only the kernel changes; the profiler shows the kernels; switching by hand with torch.backends
M07-2 Three treatments What op the same Q/DQ graph becomes through reference representation, x86 lowering, and torch.compile

The next three chapters:

Chapter Route Question it answers
M08 XNNPACK / ExecuTorch A How the two XNNPACKQuantizers differ; how a Q/DQ graph becomes a .pte; how unsupported ops are cut out; how much faster int8 is
M09 ONNX Runtime B How a Q/DQ graph is exported to QDQ ONNX; what each ORT optimization level changes; which ops become int8
M10 Custom backend Custom Writing hardware constraints as a Quantizer; checking whether a Q/DQ graph complies; falling unsupported ops back to the CPU

What this overview deliberately skips: how far the backend’s real int8 kernel is from the Q/DQ graph (M08, M09); the full OpenVINO flow (this project only did a conversion trial).

References