M07 Overview | Backend Deployment Overview: After the Q/DQ Graph
M01 to M06 all end at the Q/DQ graph produced by convert_pt2e. It marks “where to quantize and with which qparams”, but the operations themselves are still fp32 conv and linear, and it does not get faster (M02-3). To get faster, something has to replace the quantized operations on the graph with real int8 kernels. M07 to M10 answer exactly this: who does the replacing, at which step, and what to do about what cannot be replaced.
This overview has two halves:
- The first half is background shared by M07 to M10: which layers a backend is made of, which routes lead from a Q/DQ graph, and why the backend has to be chosen before quantizing.
- The second half is the M07 chapter itself: before leaving PyTorch, what else can be done with a Q/DQ graph inside PyTorch (reference representation, lowering, Inductor), and the rules to follow when comparing backend results.
The questions M07 to M10 answer
| Question | One-sentence answer | Chapter |
|---|---|---|
| Can a Q/DQ graph become integer arithmetic inside PyTorch? | An operation only becomes integer if a pattern matches; otherwise fp32 is kept | M07-2 |
| What should you know before comparing the numbers of two implementations? | Just swapping the kernel makes fp32 differ by 1e-5 to 1e-4; that is the floor | M07-1 |
| What happens when you hand it to ExecuTorch + XNNPACK? | The subgraphs are cut at export time and stored as .pte; at run time XNNPACK’s int8 kernels are used |
M08 |
| What happens when you hand it to ONNX Runtime? | Export a QDQ ONNX; ONNX Runtime only replaces the units with int8 ops when it builds the session | M09 |
| How should a Quantizer be written when the hardware has more restrictions? | Write the hardware constraints as annotations, then check the Q/DQ graph and fall back unsupported ops to the CPU | M10 |
An example first: a unit on the Q/DQ graph
The smallest conv + relu after PTQ (XNNPACKQuantizer, per-channel weight); this is the Q/DQ graph after convert:
Look at the middle part: the input is first dequantized to fp32, goes through the fp32 conv2d and relu, and is then quantized back to int8. Such a “dq → op → q” segment is called a unit:
int8 ─ dq ─→ fp32 ─ conv2d, relu ─→ fp32 ─ q ─→ int8
The weight side is the same: the int8 constant goes through dequantize_per_channel back to fp32 before entering conv2d. The unit’s semantics is int8 in, int8 out, but the operations in the middle are still fp32. To get faster, something has to recognize this unit and replace it with one int8 conv-relu kernel.
Every backend in M08 to M10 consumes a Q/DQ graph made of such units. First we look at who this graph gets handed to (first half), then at what PyTorch itself can turn the units into (second half).
From the Q/DQ graph to the hardware: three layers
ExecuTorch, ONNX Runtime, XNNPACK, and oneDNN are often lumped together as “backends”, but they are not on the same layer. Going down from the Q/DQ graph, the path is:
Q/DQ graph the endpoint of PT2E (M01 to M06)
→ deployment file .pte / .onnx / OpenVINO IR
→ inference framework ExecuTorch / ONNX Runtime / OpenVINO
→ hardware extension unit backend / Execution Provider / device plugin
→ kernel library XNNPACK / oneDNN / MLAS ...
→ hardware CPU / GPU / NPU
The three middle layers are defined as follows:
- Inference framework (often also called a runtime): reads the deployment file, decides who gets each piece of the graph, manages memory, and schedules execution. It does not necessarily compute conv itself.
- Hardware extension unit: the socket through which an inference framework connects to some hardware or library. The three frameworks name it differently: ExecuTorch calls it a backend (the subgraph handed out is a delegate), ONNX Runtime calls it an Execution Provider (EP), and OpenVINO calls it a device plugin.
- Kernel library: the code that actually computes conv and matmul. It does not read model files, so an inference framework must call it.
Inference frameworks
| ExecuTorch | ONNX Runtime (ORT) | OpenVINO (OV) | |
|---|---|---|---|
| Led by | Meta / PyTorch | Microsoft | Intel |
| Deployment file | .pte |
.onnx |
OpenVINO IR (.xml graph + .bin weights) |
| Hardware extension unit | backend | Execution Provider | device plugin |
| When it decides who gets what | At export time; the partitioner cuts the graph first | At session creation, following the EP order | At model compile time, following the specified device |
| Main target | Phones, edge devices | Cross-platform, cross-framework | Intel CPU, GPU, NPU |
The row “When it decides who gets what” is the one most worth remembering: it determines at which step the int8 kernel appears, and M08 and M09 each look at it once.
Kernel libraries
| Kernel library | Developer | Main execution target | Who calls it in this project |
|---|---|---|---|
| XNNPACK | CPUs on arm / x86 / WebAssembly | ExecuTorch’s XNNPACK backend (M08) | |
| oneDNN (formerly MKL-DNN) | Intel | Mainly x86 CPUs | PyTorch’s mkldnn, torchao’s x86 lowering (M07-2) |
| MLAS | Microsoft | CPU | ORT’s default CPU EP (M09) |
| NNPACK | Open-source project | CPU (fp32) | PyTorch eager’s conv fast path (M07-1) |
On arm you also commonly see Arm’s own ACL (Arm Compute Library) and KleidiAI. Tried on openvino 2026.4.1, macOS arm64: OpenVINO’s CPU plugin uses ACL for conv and KleidiAI for some MatMuls.
One Library, Several Frameworks
A hardware extension unit is only an adapter, and the same kernel library can be plugged in under several frameworks: XNNPACK is an ExecuTorch backend and also an ORT EP. But whether an EP is usable depends on whether the ORT build you installed has it compiled in: the pip onnxruntime, onnxruntime-gpu, and an ORT you built from source each have different EPs compiled in. ORT provides two functions to answer this:
ort.get_all_providers(): the names of all EPs this version of ORT knows about, regardless of your machine.ort.get_available_providers(): the EPs actually compiled into the build you installed and usable, in ORT’s default priority order.
When a build has several EPs, which one is used is specified at session creation by ort.InferenceSession(..., providers=[...]), with earlier entries in the list taking priority. M09 always specifies providers=["CPUExecutionProvider"], so whether or not your build has the XNNPACK EP, the int8 kernels M09 sees are always the CPU EP’s MLAS, a different implementation from M08’s XNNPACK.
On the ExecuTorch side, a backend is a subpackage under executorch.backends. List them to see (requires uv sync --group executorch):
import importlib.util
from pathlib import Path
if importlib.util.find_spec("executorch") is None:
print("executorch is not installed (uv sync --group executorch)")
else:
import executorch.backends
root = Path(executorch.backends.__path__[0])
backends = sorted(p.name for p in root.iterdir()
if p.is_dir() and p.name not in {"test", "transforms", "example", "__pycache__"})
print(f"{len(backends)} folders under executorch.backends:")
print(backends)
assert "xnnpack" in backends16 folders under executorch.backends:
['aoti', 'apple', 'arm', 'cadence', 'cortex_m', 'cuda', 'mediatek', 'mlx', 'native', 'nxp', 'openvino', 'qualcomm', 'samsung', 'vulkan', 'webgpu', 'xnnpack']
Each folder is one backend (or a group of backends): xnnpack is the CPU backend M08 uses, Core ML lives under apple, qualcomm is QNN, openvino is OpenVINO … XNNPACK is just one of them.
Behind the adapter, there is not always a kernel library
Several names are both an ExecuTorch backend and an ORT EP: OpenVINO, Core ML, QNN. They are themselves another inference framework or runtime, and the subgraph handed to them is dispatched once more:
| Name | What it is | Where it plugs in | After handing over |
|---|---|---|---|
| XNNPACK | Kernel library | ExecuTorch backend; ORT EP | Runs CPU kernels directly |
| OpenVINO | Intel’s inference framework | ExecuTorch backend; ORT EP | OpenVINO then picks the device plugin and kernel |
| Core ML | Apple’s inference framework | ExecuTorch backend; ORT EP | Core ML then picks CPU, GPU, or Neural Engine |
| QNN | Qualcomm’s runtime | ExecuTorch backend; ORT EP | QNN then picks a Qualcomm compute unit |
Who calls whom
This draws the relationship between inference frameworks and kernel libraries together (how the deployment files come about is left to the next section). Light red is an inference framework, yellow is a kernel library. Solid lines are relationships that could be verified in the versions used when writing this (package directories, EP names, source code, or a trial); dashed lines come only from documentation: OpenVINO using oneDNN on x86 is what the documentation says, and it was not tried on x86.
Three routes: starting from the Q/DQ graph
PT2E does not produce these three deployment files directly. The Q/DQ graph goes through each framework’s own bridging step to become a file the framework can read. Blue is the Q/DQ graph, green is a deployment file or model IR, light red is an inference framework; the dashed line was only tried with an fp32 model (ONNX → ov.Model):
The line from the Q/DQ graph straight to oneDNN does not go through an inference framework: torchao’s x86 lowering rewrites the Q/DQ graph directly into an fx graph that calls oneDNN (the M07 chapter proper, in the second half). As a table:
| Route A: ExecuTorch | Route B: ONNX Runtime | Route C: OpenVINO | |
|---|---|---|---|
| Starting point on the PyTorch side | The Q/DQ graph from convert_pt2e |
The Q/DQ graph from convert_pt2e |
The graph from convert_pt2e(..., fold_quantize=False) |
| Bridging step | torch.export again, then to_edge_transform_and_lower |
torch.onnx.export(..., dynamo=True) |
ov.convert_model |
| Deployment file | .pte |
QDQ ONNX (.onnx) |
ov.Model, which can be saved as OpenVINO IR |
Who recognizes dq → op → q, and when |
XNNPACK’s partitioner, at export time | ORT’s QDQ node fusion, at session creation | Not checked (q/dq first become OV’s FakeQuantize) |
| In this project | M08 | M09 | Only a conversion trial was done (see below) |
Route A fixes the kernels at export time: what is stored in the .pte is exactly the subgraph XNNPACK will execute. Route B’s .onnx is just the Q/DQ graph written in another format; turning it into int8 ops only happens when ORT builds the session, and whose kernels are used depends on the EP.
Route C: OpenVINO first converts to ov.Model
No later chapter covers OpenVINO, so here are a few things tried on openvino 2026.4.1, macOS arm64 (openvino is not a dependency of this project):
- OpenVINO does not consume PyTorch models directly:
core.compile_model(nn.Module, "CPU")fails, and you have to convert withov.convert_modelfirst. - The Q/DQ graph must be produced with
convert_pt2e(..., fold_quantize=False)to convert: the default approach stores weights as int8 constants, which gives a type error during conversion.
The result of conversion and the result of compilation are two different things:
| Term | What it is | Related to the device? |
|---|---|---|
ov.Model |
The result of ov.convert_model; the in-memory form of the OpenVINO IR |
No; can be saved and moved to another machine |
| OpenVINO IR | The .xml (graph) + .bin (weights) saved by ov.save_model |
No |
CompiledModel |
The result of core.compile_model(ov.Model, device) compiled for one device plugin |
Yes (for example "CPU") |
OpenVINO can also be embedded into the other two routes: as ORT’s OpenVINO EP, or as ExecuTorch’s openvino backend (the table in “Behind the adapter, there is not always a kernel library”).
Before Quantizing, Choose the Backend
prepare_pt2e(model, quantizer) does not choose a backend for you. The Quantizer decides which ops to quantize, which qspec to use, and which tensors share qparams (M03), and these rules are designed around some backend’s kernels:
Choose the deployment backend
→ build that backend's Quantizer decides quantization locations and qspec
→ prepare_pt2e, calibrate, convert get the Q/DQ graph
→ that backend's lowering recognizes dq → op → q, swaps in an int8 kernel
| Target backend | Matching Quantizer | In this project |
|---|---|---|
| XNNPACK | XNNPACKQuantizer (there are two with the same name; the difference is in M08-1) |
M01 to M08 |
| x86 (oneDNN) | X86InductorQuantizer |
M03-7 |
| OpenVINO | OpenVINOQuantizer (shipped with ExecuTorch’s openvino backend, built on Intel’s NNCF) |
Not used |
| A fictional VPU | The hand-written MyVPUQuantizer |
M10 |
These Quantizers all inherit from the same torchao Quantizer base class (M03); they differ in the annotation rules.
The Q/DQ graph format is backend-neutral, but the quantization locations and qspec on the graph are not. When the Quantizer and the lowering do not match, some units cannot be recognized and stay on the fallback path: the conv in the reference representation in the second half is one example (one of the mismatched conditions: the pattern needs per-tensor weight, while XNNPACKQuantizer gives per-channel). M10 stands on the backend side the other way around, and checks which constraints of the fictional VPU a Q/DQ graph violates.
The M07 chapter proper: what else a Q/DQ graph can become inside PyTorch
All three routes above leave PyTorch. Before leaving, PyTorch and torchao themselves have three ways of handling a Q/DQ graph, each answering “what happens to this unit next”.
Three terms: not a pick-one-of-three on the same layer
| Term | What it is | Provided by | Purpose |
|---|---|---|---|
| reference representation | Another output format of convert_pt2e |
torchao | Writes the integer semantics of quantization with ordinary ATen arithmetic |
| lowering | Replacing patterns on the graph with some backend’s ops | Each backend implements its own | Lets the backend run with its own kernels |
| Inductor | The default compiler of torch.compile |
PyTorch | Compiles the whole graph and generates kernels |
The reference representation is a “spec” and does not aim to be fast; lowering is the action of “swapping ops”; Inductor is a compiler, and the x86 lowering happens to borrow its pattern mechanism (see “How the three relate” below).
Reference representation: writing Q/DQ as explicit arithmetic
convert_pt2e(..., use_reference_representation=True) replaces every quantize_per_tensor / dequantize_per_tensor node on the graph with a series of ordinary ATen arithmetic nodes. Use the same conv_relu and compare the segment feeding conv2d’s input. The first figure is the Q/DQ graph, where x goes through one q and one dq into conv2d:
After switching to the reference representation, the same segment becomes 11 arithmetic nodes. It is drawn in two figures, with the constant parameters (scale, zp, dtype) under each node:
conv2d itself is unchanged: it does not match torchao’s pattern (the table below) and still consumes fp32. Only q and dq are replaced. These two series of arithmetic are the two formulas below. With scale \(s\), zero point \(z\), and integer range \([q_{min}, q_{max}]\):
\[ q = \operatorname{clamp}\big(\operatorname{round}(x / s) + z,\ q_{min},\ q_{max}\big) \tag{1} \]
\[ \hat{x} = (q - z)\cdot s \tag{2} \]
Here \(x/s\), round, and \(\times s\) are still floating-point operations: this step is “writing the formula out”, not switching to an int8 kernel.
How the operation in the middle of a unit is handled depends on whether the whole unit matches a pattern torchao has written:
| Case | The unit becomes | Example on torchao 0.18 (M07-2 B) |
|---|---|---|
| No match | int8 → dq arithmetic → fp32 operation → q arithmetic → int8 | The conv in the conv_relu above |
| Match | int8 → int32 accumulate / arithmetic → requantize → int8 | add + relu, linear with per-tensor weight |
PyTorch officially positions it as a backend-agnostic reference format (PT2E docs, RFC-0019):
- Good for: spelling out the semantics of integer accumulation, the bias scale, requantize, and clamp; serving as a common reference when backend or compiler authors implement lowering; running and checking the integer quantization logic without target hardware.
- Not good for: measuring final latency (what you measure is only the speed of the reference arithmetic); predicting the actual performance of oneDNN, XNNPACK, or an NPU; guaranteeing rounding / overflow behavior identical to specific hardware.
Lowering: the int8 op is swapped in only if the pattern matches
Lowering edits the graph at compile time; it does not ask “is there a kernel?” at run time:
decompose rearrange the graph into forms the patterns recognize (conv2d → convolution, etc.)
pattern match run the patterns the backend registered in advance, e.g. dq → conv → q
replace matched ones become the backend's op (x86: onednn.qconv_pointwise)
leave the rest anything not registered or not matched stays on the graph as is
Each backend has its own lowering:
| Backend | Lowering entry point | Units that do not match | Which chapter |
|---|---|---|---|
| x86 (oneDNN) | lower_pt2e_quantized_to_x86 |
Left as is on the fx graph, run in fp32 | M07-2 C |
| XNNPACK | to_edge_transform_and_lower(..., XnnpackPartitioner) |
Left outside the delegate, run by ExecuTorch’s portable kernels | M08 |
| ONNX Runtime | Export a QDQ ONNX; ORT does QDQ fusion when building the session | Left in fp32, with one int8 ↔︎ fp32 conversion before and after | M09 |
| A fictional VPU | The hand-written partitioner | Fall back to the CPU | M10 |
Inductor: the compiler of torch.compile
Inductor is the default backend of torch.compile. It does decompose, pattern matching, constant folding, fusion of elementwise operations, and layout and memory planning over the whole graph, and finally generates C++ kernels on the CPU; operations like conv and matmul call external libraries (oneDNN, BLAS, NNPACK, etc.). When it meets a Q/DQ graph there are two outcomes:
| Case | What Inductor does |
|---|---|
| There is an int8 pattern, and freezing is on | dq → conv → q is replaced by oneDNN’s int8 conv (same as lowering) |
| There is no int8 pattern | Elementwise operations such as dq, q, and relu are fused into the kernel it generates; conv still calls the fp32 kernel |
How the three relate
The x86 int8 patterns are not in PyTorch itself but in torchao (from reading the source of torchao 0.18 and torch 2.14):
| Step | Where | What it does |
|---|---|---|
| The pattern itself | torchao’s x86.py (inductor_passes/) |
Replaces dq → conv → q with oneDNN’s int8 op (for example qconv_pointwise) |
| Registration | When x86_inductor_quantizer is imported |
Registered only when mkldnn is available, into Inductor’s freezing patterns |
| Who runs it (1) | torchao’s lower_pt2e_quantized_to_x86 |
Calls the freezing pass directly and returns the edited fx graph |
| Who runs it (2) | torch.compile |
Runs only when Inductor’s freezing setting is on and under no_grad; freezing is off by default (torch’s compile_fx.py) |
Rules Before Comparing Backends
After M07-1, every chapter compares some implementation’s result against the Q/DQ graph or the fp32 model. Four things need to be settled before comparing.
The fp32 numeric floor
When comparing two implementations (Q/DQ graph vs. reference, eager vs. lowering, PyTorch vs. ONNX Runtime), the difference is not all caused by quantization. For a completely unquantized fp32 model, as soon as the CPU switches its conv implementation (for example, a different batch size goes through NNPACK instead), the output differs by 1e-5 to 1e-4. This magnitude is the numeric floor: a gap smaller than it is not a fault of quantization or lowering. M07-1 measures it and reproduces it on any machine with the torch.backends switches.
Numerical Differences from the Q/DQ Graph
The Q/DQ graph simulates quantization in fp32, while the backend’s integer kernels really compute with integers, and rounding happens in different places. Another known source is the bias: the bias in the Q/DQ graph is fp32, XNNPACK quantizes it to int32 during lowering with scale = \(s_x \cdot s_w\) (executorch 1.5.0’s quant_params.py, from_bias), and ORT’s ORT_ENABLE_BASIC level does the same thing (M09-2).
So when comparing backends, look only at the final output:
| Metric | What it answers |
|---|---|
| SQNR, accuracy against fp32 | Whether accuracy got worse |
| Fraction of outputs bit-identical | How close it is to the Q/DQ graph |
| argmax agreement rate | Whether the classification result changed |
| Max difference in output LSBs | Whether the difference is just rounding |
The speed baseline: fp32 in the same runtime
The Q/DQ graph itself is slower than eager (it adds fp32 q and dq operations); an fp32 model moved to another runtime can also be much faster. So the gain of int8 should be compared with the fp32 model in the same runtime (M08-4). None of M07’s three treatments produces an int8 kernel, so this chapter does not measure latency.
Deploying to a device often needs static shapes
This project’s export_module() (a wrapper in common/pt2e.py, not a torch API) enables dynamic batch by default, because calibration feeds different batch sizes. ExecuTorch’s .pte lays out memory according to the shapes at export time, so M08 uses export_module(..., dynamic_batch=False); ONNX export needs dynamic_shapes specified again (M09-1); M10’s fictional VPU only takes static shapes, and validate() rejects a dynamic batch.
Platform notes
The outputs stored in this overview and in this chapter’s notebooks were produced on macOS arm64, torch 2.14.0, torchao 0.18, executorch 1.5.0, onnxruntime 1.30.0, without mkldnn: the x86 int8 patterns are not registered, and none of M07’s three treatments turned conv into an int8 kernel. When you run it on your machine, ExecuTorch’s backend list and the notes under the tables are printed according to the measurements. The actual results of lowering and torch.compile on x86 (with mkldnn) were not verified here.
Summary and map
- Backends have three layers: the inference framework reads the deployment file and dispatches subgraphs; the hardware extension unit is an adapter; the kernel library does the computing. The same name may appear on several layers.
- All three routes start from the same Q/DQ graph; they differ in the bridging step and in “who, at which step” replaces
dq → op → qwith an int8 kernel. - The Quantizer must match the backend: the quantization locations and qspec are designed around the backend’s kernels, and when they do not match, units stay on the fallback path.
- Inside PyTorch, the reference representation writes out the math of quantization, and only units that match a pattern become integer; lowering replaces matched patterns with the backend’s int8 ops; Inductor compiles the whole graph and, with freezing on and a matching pattern, also swaps in int8 kernels.
- Before comparing backends, know the fp32 floor, compare only the final outputs, and compare speed against fp32 in the same runtime.
The notebooks of this chapter:
| notebook | Question it answers |
|---|---|
| M07-1 The fp32 numeric floor | How much fp32 differs when only the kernel changes; the profiler shows the kernels; switching by hand with torch.backends |
| M07-2 Three treatments | What op the same Q/DQ graph becomes through reference representation, x86 lowering, and torch.compile |
The next three chapters:
| Chapter | Route | Question it answers |
|---|---|---|
| M08 XNNPACK / ExecuTorch | A | How the two XNNPACKQuantizers differ; how a Q/DQ graph becomes a .pte; how unsupported ops are cut out; how much faster int8 is |
| M09 ONNX Runtime | B | How a Q/DQ graph is exported to QDQ ONNX; what each ORT optimization level changes; which ops become int8 |
| M10 Custom backend | Custom | Writing hardware constraints as a Quantizer; checking whether a Q/DQ graph complies; falling unsupported ops back to the CPU |
What this overview deliberately skips: how far the backend’s real int8 kernel is from the Q/DQ graph (M08, M09); the full OpenVINO flow (this project only did a conversion trial).
References
- PyTorch official tutorial: PyTorch 2 Export Post Training Quantization (the positioning of the reference representation)
- RFC-0019: Extending PyTorch Quantization to Custom Backends
- torchao 0.18 source:
reference_representation_rewrite.py,inductor_passes/x86.py,lowering.py - torch 2.14 source:
_inductor/compile_fx.py(when freezing runs); docs: torch.compiler - ExecuTorch: Backends overview;
backends/in the 1.5.0 source - ONNX Runtime: Execution Providers; v1.30.0 source
- XNNPACK; OpenVINO docs: Model Preparation (
ov.convert_model) - All About Quantization (in Chinese): derives pure integer arithmetic from the affine quantization formula, including requantize using only integer multiplication and shifts (fractional multiplication), which corresponds to the integer arithmetic in the reference representation