model = get_model("conv_relu").eval()
x = example_inputs(1) # batch 1
gm = torch.export.export(model, x).module() # static shape
quantizer = XNNPACKQuantizer().set_global(get_symmetric_quantization_config(is_per_channel=True))
prepared = prepare_pt2e(gm, quantizer)
with torch.no_grad():
for (batch,) in calibration_batches(32):
for i in range(len(batch)):
prepared(batch[i : i + 1])
qdq = convert_pt2e(prepared) # Q/DQ graphM08 Overview | XNNPACK / ExecuTorch: Turning the Q/DQ Graph into a .pte
All three treatments in M07 stay inside PyTorch: the Q/DQ graph goes through the reference representation, x86 lowering, and torch.compile, and conv is still fp32 (in an environment without mkldnn). This chapter takes Route A from the M07 overview: hand the Q/DQ graph to ExecuTorch’s XNNPACK backend, fix the int8 kernels at export time, and save a deployment file, the .pte.
Which layer ExecuTorch and XNNPACK each sit at (inference framework, hardware extension unit, kernel library) was covered in the first half of the M07 overview, so it is not repeated here. This chapter answers four questions:
| Question | One-line answer | notebook |
|---|---|---|
Which XNNPACKQuantizer should I use? |
There are two with the same name; use the executorch one to export a .pte |
M08-1 |
How does a Q/DQ graph become a .pte? |
to_edge_transform_and_lower carves out the delegate, to_executorch serializes |
M08-2 |
| What if XNNPACK does not support an op? | It stays outside the delegate and is run by ExecuTorch; the graph is split | M08-3 |
| Is it actually faster? | Switching runtime is already much faster; int8 must be compared against fp32 on the same runtime | M08-4 |
This chapter needs executorch (uv sync --group executorch).
A First Example: conv + relu Through the Whole Pipeline
The smallest conv + relu, from a PyTorch model all the way to execution in the ExecuTorch runtime. Below uses only the raw PyTorch, torchao, and ExecuTorch APIs, in four steps.
Step 1: PT2E, get the Q/DQ graph. Same flow as from M01 on, with two differences: the quantizer is the one shipped with ExecuTorch’s XNNPACK backend (M08-1); and torch.export.export is not given dynamic_shapes, so the shape is fixed to batch 1 from x (the reason is in M08-2 C), which means the calibration data is fed one sample at a time.
This is the “dq → op → q” unit from the start of the M07 overview: int8 in, int8 out, with conv and relu computed in fp32 in between.
Step 2: lowering. Export once more (to_edge_transform_and_lower takes an ExportedProgram, while convert_pt2e gives a GraphModule), then let XnnpackPartitioner carve out the subgraph XNNPACK can handle:
edge = to_edge_transform_and_lower(torch.export.export(qdq, x), partitioner=[XnnpackPartitioner()])
lowered = edge.exported_program().graph_moduleconv, relu, and q/dq are all gone, replaced by a single executorch_call_delegate. The lowered_module_0 it calls is XNNPACK’s delegate, which keeps the original subgraph it absorbed:
delegate = getattr(lowered, "lowered_module_0")
inner = delegate.original_module.graph_module # the subgraph absorbed into the delegate
print("backend:", delegate.backend_id)backend: XnnpackBackend
The whole unit, including the weight’s dequantize_per_channel, is inside the delegate. The names also changed: conv2d has already been decomposed into convolution in the Edge dialect.
Step 3: serialize to a .pte. to_executorch() does memory planning for the part of the graph outside the delegate, and .buffer is the bytes of the .pte file:
pte = edge.to_executorch().buffer
print(f".pte: {len(pte)} bytes").pte: 3872 bytes
What you hand over at deployment is these bytes (written out as a model.pte file).
Step 4: run in the ExecuTorch runtime. The first time the XNNPACK delegate is loaded, ExecuTorch’s C++ layer prints a few CPU-detection log lines to stderr. They have nothing to do with the result; this project’s capture_stderr() (common/env.py, not an executorch API) swallows them:
with capture_stderr():
program = Runtime.get().load_program(pte) # a path to a .pte file also works
forward = program.load_method("forward")
assert forward is not None
y = forward.execute([x[0]])[0]
print("output shape:", tuple(y.shape))output shape: (1, 8, 16, 16)
program must be kept alive during execution: if you keep only forward and let program be garbage collected, forward reads freed memory when it runs.
The later notebooks use three wrappers from this project (common/executorch_utils.py, none of them executorch APIs):
| Wrapper | What it does |
|---|---|
xnnpack_ptq(model, x) |
Step 1: static-shape export, executorch’s quantizer, calibration, convert |
lower_to_xnnpack(gm, x) |
Steps 2 and 3, returns (edge, pte) |
PteRunner(pte) |
Step 4, and keeps program alive |
Terms
| Term | What it is |
|---|---|
| Edge dialect | ExecuTorch’s intermediate representation: most ops are further decomposed into Core ATen ops (e.g. conv2d → convolution); the partitioner can ask to keep ops it recognizes undecomposed (e.g. linear; see config_partitioner.py in executorch 1.5.0) |
| partitioner | Finds the connected subgraphs a given backend can handle; here, XnnpackPartitioner |
| delegate | A piece of the graph handed to a backend, which compiles it into its own format at export time; an executorch_call_delegate is left on the graph |
| memory planning | to_executorch() decides ahead of time, from the shapes at export time, which block of memory each intermediate tensor goes in, so nothing is allocated at run time |
.pte |
ExecuTorch’s deployment file: the graph, weights, delegate data, and memory plan are all inside |
| portable kernel | ExecuTorch’s own generic C++ op implementations, which run the ops that did not go into a delegate |
Where q/dq Go During Lowering
All the q/dq on the Q/DQ graph go into the delegate, but not every one becomes an operation. XNNPACK handles them in two ways (see backends/xnnpack/operators/op_quant_dequant.py and quant_params.py in executorch 1.5.0):
| On the Q/DQ graph | In XNNPACK |
|---|---|
| q, dq sandwiched between two ops | No operation is produced; scale and zero point are just recorded on the int8 tensor (QuantParams.from_q_dq_node). One unit is one int8 kernel (M08-2 A prints the nodes inside the delegate) |
| q at the graph input, dq at the output | Become fp32 ↔︎ int8 conversions (XNNPACK’s Convert) |
| weight (int8 constant + per-channel scale) | Stored into the .pte as is |
| fp32 bias | Quantized to int32 with scale = \(s_x \cdot s_w\) (QuantParams.from_bias) |
| ops the partitioner does not take | Stay outside the delegate and run on portable kernels (M08-3) |
So the scales XNNPACK uses are exactly the ones convert_pt2e wrote on the q/dq (the observer’s calculate_qparams()). A QAT model is the same: deployment uses the scales on the converted graph, not the fake-quant buffers from training (M06-3).
Numerical Differences from the Q/DQ Graph
Both sides get the same scale and zero point, but the outputs are still not guaranteed to be bit-identical: the Q/DQ graph simulates quantization in fp32, while XNNPACK computes with integer kernels, and rounding happens at different places.
- One known cause is the bias: on the Q/DQ graph the bias is fp32, and XNNPACK quantizes it to int32 during lowering (table above).
- Other possible sources are the integer implementations of add and mean, and how conv accumulates (inference; I did not read XNNPACK’s kernel source).
These differences come from integer rounding: on executorch 1.5.0 and macOS arm64, the largest difference measured in M08-2 and M08-3 is 1 output LSB. So accuracy evaluated on the Q/DQ graph (M05, M06) is representative of the deployed result; just do not expect bit-identical outputs. For the comparison metrics, follow “Rules Before Comparing Backends” in the M07 overview.
Summary and Chapter Map
- Route A decides the kernels at export time:
to_edge_transform_and_lowerhas the partitioner carve out the delegate, and what the.ptestores is the subgraph XNNPACK will execute. - q/dq between ops leave only the int8 tensor’s scale; only the graph’s input and output really convert fp32 ↔︎ int8.
- The
.ptedoes memory planning from the shapes at export time, so this chapter uses static shapes throughout.
| notebook | Question it answers |
|---|---|
| M08-1 Two XNNPACKQuantizers | How the executorch version differs from the torchao test copy; static vs dynamic specs; set_module_name |
M08-2 From Q/DQ Graph to .pte |
Graphs before and after lowering, XNNPACK nodes inside the delegate, .pte size, runtime execution, what happens with a different batch |
| M08-3 Unsupported Ops | How the graph is split, what crosses the boundary, how far it is from the Q/DQ graph |
| M08-4 latency | Which is faster among eager, Q/DQ graph, fp32 .pte, and int8 .pte; the gain splits into two parts |
Deliberately skipped in this chapter: .pte with dynamic shapes; ExecuTorch backends other than XNNPACK (the ones listed in the M07 overview); ONNX Runtime’s Route B is in M09.
References
- ExecuTorch docs: Backends overview, XNNPACK backend
- executorch 1.5.0 source:
backends/xnnpack/operators/op_quant_dequant.py,quant_params.py,backends/xnnpack/quantizer/xnnpack_quantizer.py - XNNPACK