import copy
from collections import Counter
import torch
import torch.nn.functional as F
from common.graph_utils import qdq_nodes, short
from common.models import ConvBnReLU, example_inputs
from common.pt2e import convert_pt2e, export_module, get_xnnpack_quantizer, prepare_pt2e, prepare_qat_pt2e, quiet
from common.viz import draw, table
quiet()
_ = torch.manual_seed(0)M04-2 | conv + BN in QAT: BN Stays, and a Separate Folded Copy Is Made for Weight Quantization
PTQ folds BN into conv at prepare_pt2e (M04-1). QAT cannot do this: during training BN must keep using the current batch’s statistics, and if it is folded there is no BN left to train. But we also want the weight’s fake quant to quantize the “folded weight”, so that the quantization error seen in training matches deployment.
prepare_qat_pt2e does this by inserting a small subgraph around the conv and BN. The full reasoning for why it is inserted this way is in the section “QAT: Which Weight Should the Weight’s Fake Quant Quantize” of the M04 overview, sourced from the blog post “Convolution and BatchNorm Fusion” (in Chinese); this notebook maps that reasoning onto the graph.
- A. For the same conv_bn_relu: how the graphs from
prepare_pt2eandprepare_qat_pt2ediffer. - B. Reading the QAT subgraph: why it is inserted this way and what each extra op computes.
- C. After
convert_pt2e: QAT’s BN is folded too; and when folding fails.
As in M04-1, first give BN real running stats. QAT must export with the training IR (export_module(..., train=True)) so that BN is the training version of the op; the quantizer must be passed is_qat=True so that fake quant is what gets inserted:
model = ConvBnReLU(3, 8).train()
with torch.no_grad():
for _ in range(5):
model(torch.randn(16, 3, 16, 16) * 2 + 1)
ptq = prepare_pt2e(export_module(copy.deepcopy(model).eval(), example_inputs()),
get_xnnpack_quantizer(is_per_channel=True)[0])
qat = prepare_qat_pt2e(export_module(copy.deepcopy(model), example_inputs(), train=True),
get_xnnpack_quantizer(is_per_channel=True, is_qat=True)[0])A. How the Graphs from prepare_pt2e and prepare_qat_pt2e Differ
Which ops each graph has and how many (call_module is written as the module class):
def op_counter(g):
return Counter(type(g.get_submodule(n.target)).__name__ if n.op == "call_module" else short(n.target)
for n in g.graph.nodes if n.op in ("call_function", "call_module") and n.name != "_guards_fn")
c_ptq, c_qat = op_counter(ptq), op_counter(qat)
table([[k, c_ptq.get(k, 0), c_qat.get(k, 0)] for k in sorted(set(c_ptq) | set(c_qat))], ["op", "prepare_pt2e", "prepare_qat_pt2e"])| op | prepare_pt2e | prepare_qat_pt2e |
|---|---|---|
| FusedMovingAvgObsFakeQuantize | 0 | 3 |
| HistogramObserver | 2 | 0 |
| PerChannelMinMaxObserver | 1 | 0 |
| add | 0 | 1 |
| add_ | 0 | 1 |
| batch_norm | 0 | 1 |
| conv2d | 1 | 1 |
| div | 0 | 2 |
| mul | 0 | 1 |
| relu | 1 | 1 |
| reshape | 0 | 2 |
| sqrt | 0 | 1 |
assert c_ptq["batch_norm"] == 0 and c_qat["batch_norm"] == 1
assert c_qat["FusedMovingAvgObsFakeQuantize"] == 3- PTQ:
batch_normis already folded; the quantization points are three observers (they only collect statistics and do not change values). - QAT:
batch_normis still there, with the extra chain of opsmul,div,sqrt,reshapethat computes the BN scale; the quantization points are threeFusedMovingAvgObsFakeQuantize(which really fake-quantize). The “fused” in its name has nothing to do with this chapter’s conv + BN; it means that “update min/max, compute scale/zero point, quantize → dequantize” are merged into one ATen kernel. The way this kernel computes scale has a pitfall, handled in M06-3. - The quantization points are in the same places: input
x, the weight, and the relu output. What differs is what the weight’s one quantizes; see section B. add_is BN’snum_batches_tracked += 1, present in every train-mode export (M00-2).
B. Reading the QAT Subgraph
First a recap of the overview’s conclusions (\(\sigma\) is the standard deviation with \(\varepsilon\) included; subscript \(B\) means this batch’s statistics, no subscript means running stats):
- The weight’s fake quant should quantize \(W_{\text{inf}} = \gamma W / \sigma\): it is the weight actually quantized at deployment, and the running stats change slowly, so the quantization grid does not jump every batch. If it quantized \(W_{\text{train}} = \gamma W / \sigma_B\) instead, the grid would jump with \(\sigma_B\) and training would be unstable.
- BN must still train with batch statistics. The whitepaper’s “slow version” computes an extra unquantized conv to get \(\mu_B\), \(\sigma_B\); PyTorch’s default “approximate version”
_forward_approximatesaves that conv, andprepare_qat_pt2euses it:
\[ s = \frac{\gamma}{\sigma}, \qquad y = \mathrm{BN}_{\text{batch}}\!\Big( \frac{\mathrm{conv}\big(\mathrm{fq}(x),\ \mathrm{fq}(W \cdot s)\big)}{s} \Big) \tag{1} \]
Draw the whole QAT graph, with each term of Eq. (1) annotated under its node:
nodes = {n.name: n for n in qat.graph.nodes}
fq_w = next(n for n in qat.graph.nodes if n.op == "call_module" and n.args[0].target == torch.ops.aten.mul.Tensor)
notes = {
"add": "running_var + ε",
"add_": "num_batches_tracked += 1(M00-2), unrelated to quantization",
"div": "s = γ / σ(σ is the running std)",
"reshape": "s reshaped to [C,1,1,1]: aligns with the weight's output channel",
"mul": "W · s = γW/σ = W_inf",
fq_w.name: "fq(W_inf): quantizes the weight used at deployment",
"reshape_1": "s reshaped to [1,C,1,1]: aligns with the conv output's channel",
"div_1": "÷ s: recovers approximately the unfolded conv output",
"batch_norm_1": "BN uses this batch's statistics μ_B', σ_B'",
}
draw(qat, notes=notes, title="conv → bn → relu after prepare_qat_pt2e")assert nodes["batch_norm_1"].args[0] is nodes["div_1"] # BN takes the conv output after ÷ s
assert fq_w.args[0] is nodes["mul"] # the weight's fake quant takes W · sReading from top to bottom:
add→sqrt→divcomputes the per-channel \(s = \gamma/\sigma\) from the running var.reshape→mulcomputes \(W \cdot s = W_{\text{inf}}\) and hands it to the weight’s fake quant. The conv itself has no bias; BN’s \(\beta\) is added later.- The conv output is computed with the folded weight, so it is “already multiplied by \(s\)”.
reshape_1→div_1divides by \(s\) again, recovering approximately the output of the unfolded conv. - After recovery, the following
batch_normworks as if nothing were quantized: it computes this batch’s mean \(\mu_B'\) and standard deviation \(\sigma_B'\) from its input to normalize, and updates the running stats as usual.
Why divide by \(s\) in step 3? Expand the BN in step 4 (\(z\) is the output of the quantized conv):
\[ \gamma\,\frac{z/s - \mu_B'}{\sigma_B'} + \beta = z \cdot \frac{\sigma}{\sigma_B'} + \Big(\beta - \frac{\gamma\,\mu_B'}{\sigma_B'}\Big) \tag{2} \]
The right-hand side is exactly the slow version’s formula: the conv with quantized \(W_{\text{inf}}\), multiplied by the correction term \(\sigma/\sigma_B'\), plus the bias from batch statistics. Dividing by \(s\) lets BN compute the correction term itself, so no extra unquantized conv is needed; the cost is that \(\mu_B'\), \(\sigma_B'\) come from the quantized conv output, not the unquantized one the slow version uses.
Verify that both sides of Eq. (2) are equal, using the fake quant modules and parameters in the graph:
params = dict(qat.named_parameters())
fq_x_mod = copy.deepcopy(qat.get_submodule(nodes["conv2d_1"].args[0].target)) # a copy: fake quant updates its state during forward
fq_w_mod = copy.deepcopy(qat.get_submodule(fq_w.target))
w, gamma, beta = params["conv.weight"], params["bn.weight"], params["bn.bias"]
sigma = torch.sqrt(qat.get_buffer("bn.running_var") + 1e-5) # running std, including ε (BatchNorm2d default 1e-5)
s = gamma / sigma
x = torch.randn(16, 3, 16, 16) * 2 + 1
with torch.no_grad():
z = F.conv2d(fq_x_mod(x), fq_w_mod(w * s.reshape(-1, 1, 1, 1)), None, padding=1) # output of the quantized conv
u = z / s.reshape(1, -1, 1, 1)
left = F.batch_norm(u, None, None, gamma, beta, training=True, eps=1e-5) # left side of Eq. (2): what the graph does
mu_b, sigma_b = u.mean((0, 2, 3)), torch.sqrt(u.var((0, 2, 3), unbiased=False) + 1e-5)
right = z * (sigma / sigma_b).reshape(1, -1, 1, 1) + (beta - gamma * mu_b / sigma_b).reshape(1, -1, 1, 1) # right side
print(f"max difference between the two sides of Eq. (2): {float((left - right).abs().max()):.1e}")
assert torch.allclose(left, right, atol=1e-5)max difference between the two sides of Eq. (2): 4.8e-07
The two sides differ only by floating-point error: the approximate version is the slow version “with quantized batch statistics”. How much they differ numerically is the topic of M04-3.
C. After convert_pt2e: QAT’s BN Is Folded Too
convert_pt2e has no parameter like is_qat; PTQ and QAT take the same path: first replace observers/fake quants with Q/DQ, then try to fold BN (torchao’s internal _fold_conv_bn_qat; the M02 overview listed the steps of convert):
- No BN in the graph (PTQ already folded it at prepare): skip.
- BN present: match the pattern “the conv + BN subgraph produced by
prepare_qat_pt2e+ Q/DQ”, and fold only if the match succeeds. If the match fails, BN is left as is, with no error and no warning.
The third path is a deliberately wrong demo: prepare_qat_pt2e paired with a quantizer with is_qat=False.
wrong = prepare_qat_pt2e(export_module(copy.deepcopy(model), example_inputs(), train=True),
get_xnnpack_quantizer(is_per_channel=True, is_qat=False)[0])
rows = []
for name, g in [("PTQ", ptq), ("QAT", qat), ("QAT, but quantizer with is_qat=False", wrong)]:
converted = convert_pt2e(copy.deepcopy(g))
counts = op_counter(converted)
folded_bias = any("weight_bias" in str(n.target) for n in converted.graph.nodes if n.op == "get_attr")
rows.append([name, counts.get("batch_norm", 0), counts.get("conv2d", 0), counts.get("relu", 0),
len(qdq_nodes(converted)), folded_bias])
table(rows, ["path", "batch_norm", "conv2d", "relu", "# q/dq", "has folded bias"])| path | batch_norm | conv2d | relu | # q/dq | has folded bias |
|---|---|---|---|---|---|
| PTQ | 0 | 1 | 1 | 5 | True |
| QAT | 0 | 1 | 1 | 5 | True |
| QAT, but quantizer with is_qat=False | 1 | 1 | 1 | 5 | False |
assert [r[1] for r in rows] == [0, 0, 1]
assert rows[0][2:] == rows[1][2:] # PTQ and QAT have the same shape after convert- First two rows: after convert, the QAT graph really has BN folded into conv, and like PTQ is left with only conv2d, relu, and q/dq. The graph the backend receives does not know, and does not need to know, which path was taken.
- Third row: a quantizer with
is_qat=Falseinserts plain observers, not fake quants, so the subgraph structure differs and BN remains in the graph after convert. Inference (the exact step where the match fails was not traced here): the pattern structure does not match. A FakeQuantize + conv + BN graph assembled by hand is the same: if it does not match a supported pattern, it cannot be folded.
Summary
- QAT’s BN is not folded at prepare; it stays and trains with batch statistics.
prepare_qat_pt2einserts a separate subgraph so that the weight’s fake quant quantizes the folded weight \(W \cdot s\). convert_pt2edoes the real BN folding, and the final graph has the same shape as PTQ’s.- Whether folding works depends on the graph structure produced by prepare (including the quantizer’s
is_qat), not on a convert parameter.