M09-1 | From Q/DQ Graph to QDQ ONNX: What torch.onnx.export Turns Each Op Into
Open In Colab
M08 handed the Q/DQ graph to ExecuTorch; this chapter takes Route B (M07 overview): export to .onnx first, then hand it to ONNX Runtime (ORT). This notebook covers only the first step, the export:
PT2E Q/DQ graph (torchao test-version XNNPACKQuantizer)
→ torch.onnx.export(dynamo=True) ONNX ModelProto (saving it gives the .onnx)
A. What the ONNX graph looks like: which ONNX op each PT2E op becomes.
B. What the file stores: int8 weights, per-channel scales, fp32 bias; the size of the .onnx.
C. Dynamic batch: the dynamic batch on a GraphModule is not carried into ONNX automatically.
The model is M05’s TinyConvNet (trained). xnnpack_qdq is a wrapper from this project (common/onnx_utils.py, not a torchao API) that does exactly the PTQ of M05-1: export_module (dynamic batch) → prepare_pt2e (torchao test-version XNNPACKQuantizer, per-channel weight) → calibration → convert_pt2e.
The export itself is one line, torch.onnx.export(..., dynamo=True). It re-runs torch.export on x, then translates each ATen op into an ONNX op; what dynamic_shapes is for is covered in section C.
This project wraps that line as to_onnx(gm, x) (common/onnx_utils.py), which the later notebooks use.
A. What the ONNX Graph Looks Like
Start with the first conv unit. draw_onnx is this project’s tool for drawing ONNX graphs (common/viz.py), used like the draw that drew fx graphs in earlier chapters: the second line of a node is the ONNX node’s name, ‹…› holds constants stored in the file, and scale and zero point are printed under Q/DQ.
draw_onnx(onnx_model, focus="node_conv2d", hops=2, title="QDQ ONNX: first conv unit")
It is identical to the Q/DQ graph seen in M01, just renamed: quantize_per_tensor → QuantizeLinear, dequantize_per_tensor → DequantizeLinear, conv2d → Conv. The DequantizeLinear(int8[…]) on the weight side takes only constants, so it is folded into the Conv’s label.
The correspondence for the whole graph:
PT2E_TO_ONNX = [ ("quantize_per_tensor", "QuantizeLinear", "activation fp32 → int8"), ("dequantize_per_tensor", "DequantizeLinear", "activation int8 → fp32"), ("dequantize_per_channel", "DequantizeLinear", "weight int8 → fp32; axis=0: one scale per output channel"), ("conv2d", "Conv", ""), ("relu", "Relu", ""), ("max_pool2d", "MaxPool", ""), ("add", "Add", ""), ("adaptive_avg_pool2d", "ReduceMean", "adaptive pool with 1×1 output equals the mean over H, W"), ("flatten", "Reshape", ""), ("linear", "Gemm", ""),]pc, oc = op_counts(qdq), onnx_op_counts(onnx_model)rows = [[p, pc.get(p, 0), o, oc.get(o, 0), note] for p, o, note in PT2E_TO_ONNX]rows[2][2:4] = ["↑ (same as above)", ""] # both kinds of dq become DequantizeLinear; the combined count is in the previous rowtable(rows, ["Q/DQ graph op", "Count", "ONNX op", "Count", "Note"])
Q/DQ graph op
Count
ONNX op
Count
Note
quantize_per_tensor
11
QuantizeLinear
11
activation fp32 → int8
dequantize_per_tensor
12
DequantizeLinear
16
activation int8 → fp32
dequantize_per_channel
5
↑ (same as above)
weight int8 → fp32; axis=0: one scale per output channel
conv2d
4
Conv
4
relu
4
Relu
4
max_pool2d
2
MaxPool
2
add
1
Add
1
adaptive_avg_pool2d
1
ReduceMean
1
adaptive pool with 1×1 output equals the mean over H, W
flatten
1
Reshape
1
linear
1
Gemm
1
Each op becomes a standard ONNX op one-to-one, and the Q/DQ positions are unchanged. This style is called the QDQ format: all quantization information sits on QuantizeLinear / DequantizeLinear, while the compute ops themselves are still the fp32 Conv and Gemm.
Only one number does not match: there are fewer DequantizeLinear than the two kinds of dq in the Q/DQ graph added together. Look near the residual Add:
draw_onnx(onnx_model, focus=lambda n: n.op_type =="Add", hops=2, title="QDQ ONNX: around the residual Add")
n_dq = pc["dequantize_per_tensor"] + pc["dequantize_per_channel"]print(f"dq in the Q/DQ graph: {n_dq}; DequantizeLinear in the QDQ ONNX: {oc['DequantizeLinear']}")assert oc["DequantizeLinear"] < n_dq
dq in the Q/DQ graph: 17; DequantizeLinear in the QDQ ONNX: 16
The DequantizeLinear on the right feeds both the Add and the next Conv. From the Q/DQ graph to ORT, this dq is duplicated, merged, and duplicated again, and the three steps are done by three different tools:
Step
Why
Duplicate: DuplicateDQPass in convert_pt2e (M02-3) makes one copy for each user
The Q/DQ graph does not know which backend it will go to, so it is written uniformly with a complete dq → op → q around every op, letting the backend match unit by unit and swap in int8 kernels
A generic cleanup (onnxscript’s CommonSubexpressionEliminationPass) that knows nothing about QDQ: two nodes with the same op and inputs are treated as duplicate computation
Duplicate again: when ORT creates the session (M09-2)
ORT also looks for kernels in units of DQ → op → Q, so before any graph optimization it uses EnsureUniqueDQForNodeUnit to give each unit its own DQ back
The party that needs “each unit has its own dq” is the one looking for kernels; the merge in between is just the export tool not knowing about this need.
To confirm in code that the merge comes from torch.onnx.export’s graph optimization: turn it off with torch.onnx.export(Q/DQ graph, x, dynamo=True, optimize=False) (to_onnx(..., optimize=False)), and the number of DequantizeLinear equals the number of dq in the Q/DQ graph:
n_raw = onnx_op_counts(to_onnx(qdq, x, optimize=False))["DequantizeLinear"]print(f"With torch.onnx.export graph optimization off: {n_raw} DequantizeLinear ({n_dq} dq in the Q/DQ graph)")assert n_raw == n_dq
With torch.onnx.export graph optimization off: 17 DequantizeLinear (17 dq in the Q/DQ graph)
The two merged dq have identical inputs and parameters, so the output is the same whether or not they are merged; ORT splits them again by itself, so the rest of this chapter exports with the default optimize=True as usual.
B. What the File Stores
ONNX stores constants as initializers. Grouped by purpose:
def role(op_type, idx):"""The initializer is input number idx of some op -> its purpose."""if op_type =="DequantizeLinear":return ["weight (int8)", "weight per-channel scale", "weight zero_point"][idx]if op_type in ("Conv", "Gemm") and idx ==2:return"conv/linear bias"return {"ReduceMean": "ReduceMean axes", "Reshape": "Reshape target shape"}.get(op_type, f"{op_type} parameter")roles = {}for n in onnx_model.graph.node:for idx, name inenumerate(n.input): roles.setdefault(name, role(n.op_type, idx))groups, n_scalar = {}, 0for init in onnx_model.graph.initializer: arr = numpy_helper.to_array(init)if arr.size ==1: n_scalar +=1continue g = groups.setdefault(roles[init.name], [str(arr.dtype), 0, []]) g[1] +=1 g[2].append("×".join(map(str, arr.shape)))rows = [[r, dt, cnt, ", ".join(shapes[:3]) + ("…"iflen(shapes) >3else"")] for r, (dt, cnt, shapes) in groups.items()]rows.append(["activation scale/zero_point", "float32/int8", n_scalar, "scalar"])table(rows, ["Purpose", "dtype", "Count", "shape"])
The int8 weights are constants that convert_pt2e has already quantized (M02-3), written into the file as they are; one scale per output channel.
The weights are symmetrically quantized, so every zero point is 0; zero points with identical content are stored once (shared by weights with the same channel count), so there are fewer of them than weights.
The bias is not quantized in the Q/DQ graph, so it is still float32 in the file.
The file holds int8 weights, but the file itself contains no int8 kernel: Conv is still an fp32 op, and whether it is replaced by an int8 kernel is decided by the ORT that runs it (M09-2).
Export the fp32 model too, and compare the sizes of the two files piece by piece:
def sizes(name, m):"""(total .onnx size, initializer, node, of which node metadata_props), in bytes.""" init =sum(len(i.SerializeToString()) for i in m.graph.initializer) nodes =sum(len(n.SerializeToString()) for n in m.graph.node) meta =sum(len(p.SerializeToString()) +2for n in m.graph.node for p in n.metadata_props)return [name, len(m.SerializeToString()), init, nodes, meta]rows = [sizes("torch.onnx.export(fp32 model) (to_onnx)", to_onnx(model, x)), sizes("torch.onnx.export(Q/DQ graph)", onnx_model)]table(rows, ["", "Total", "initializer", "node", "metadata_props in nodes"])
Total
initializer
node
metadata_props in nodes
torch.onnx.export(fp32 model) (to_onnx)
58641
39795
15133
13435
torch.onnx.export(Q/DQ graph)
64495
11112
46108
40989
init_ratio, total_ratio = rows[1][2] / rows[0][2], rows[1][1] / rows[0][1]print(f"Initializers are {init_ratio:.0%} of fp32; total file size is {total_ratio:.0%} of fp32")if total_ratio >1:print("The file actually grows: the extra QuantizeLinear/DequantizeLinear nodes each carry their own metadata_props.")assert init_ratio <0.5
Initializers are 28% of fp32; total file size is 110% of fp32
The file actually grows: the extra QuantizeLinear/DequantizeLinear nodes each carry their own metadata_props.
Weights go from 4 bytes to 1 byte, so the initializers shrink a lot. But torch.onnx.export writes provenance information (the original fx node, the module path) into each node’s metadata_props, and Q/DQ adds a pile of nodes; in this small model the nodes take more space than the weights. The larger the model, the higher the weights’ share (an inference; no larger model was measured).
C. Dynamic Batch
The first step of the Q/DQ graph is torch.export.export(..., dynamic_shapes=({0: Dim.AUTO},)) (wrapped in export_module), so the batch dimension is dynamic. Export the same Q/DQ graph twice, differing only in whether dynamic_shapes is passed; create one ort.InferenceSession for each (ort_session) and feed two batch sizes:
rows, dim0 = [], {}for dynamic in [False, True]: m = to_onnx(qdq, x, dynamic_batch=dynamic) d = m.graph.input[0].type.tensor_type.shape.dim[0] dim0[dynamic] = d.dim_param or d.dim_value sess = ort_session(m) results = []for bs in [x[0].shape[0], 4]:try: ort_run(sess, torch.randn(bs, 3, 16, 16)) results.append(f"batch {bs} OK")exceptExceptionas e: # ORT's shape error is InvalidArgument results.append(f"batch {bs}{type(e).__name__}") rows.append(['dynamic_shapes=({0: Dim("batch")},)'if dynamic else"no dynamic_shapes", dim0[dynamic], "; ".join(results)])table(rows, ["torch.onnx.export(Q/DQ graph, x, dynamo=True, …)", "Input dim 0", "ORT run"])
torch.onnx.export(dynamo=True) re-runs torch.export on x, so without dynamic_shapes dimension 0 is hard-coded to x’s batch, and the GraphModule’s original dynamic batch is not carried over. With dynamic_shapes, dimension 0 becomes a symbol and ORT accepts any batch. The rest of this chapter exports this way (to_onnx defaults to dynamic_batch=True).
Summary
After export to ONNX the Q/DQ graph keeps its structure: q/dq become QuantizeLinear / DequantizeLinear, and the other ops become standard ONNX ops. This is the QDQ format.
In the file, weights are int8 + per-channel scale and the bias is still fp32; the file contains no int8 kernel yet.
Dynamic batch has to be stated again in torch.onnx.export.
Whether “DequantizeLinear → Conv → QuantizeLinear” is replaced by an int8 kernel is decided by the ORT that runs it (M09-2).