import copy
import torch
import torch.nn.functional as F
from common.graph_utils import sqnr
from common.models import ConvBnReLU, example_inputs
from common.pt2e import export_module, get_xnnpack_quantizer, prepare_qat_pt2e, quiet
from common.viz import table
quiet()
_ = torch.manual_seed(0)
EPS = 1e-5 # BatchNorm2d default εM04-3 | Slow vs. Approximate Version: Which One PT2E’s QAT conv + BN Uses
M04-2 looked at the subgraph inserted by prepare_qat_pt2e. The M04 overview explained that in QAT the weight’s fake quant should quantize \(W_{\text{inf}} = \gamma W/\sigma\), and that there are two ways to make BN still use batch statistics, both summarized in the blog post “Convolution and BatchNorm Fusion” (in Chinese):
| Approach | Source | Difference |
|---|---|---|
| Slow version | The correction-term approach of the quantization whitepaper (R. Krishnamoorthi, 2018); PyTorch’s _forward_slow |
BN’s batch statistics come from the unquantized conv output, so one extra conv is needed |
| Approximate version | PyTorch eager QAT’s default _forward_approximate (conv_fused.py) |
Batch statistics come from the quantized conv output, saving that extra conv |
This notebook hand-writes both with torch.nn.functional and compares them with the PT2E graph:
- A. Hand-write the approximate version and compare it bit for bit with the graph from
prepare_qat_pt2e. - B. Hand-write the slow version and see how much it differs from PT2E.
- C. Where the difference lies.
Build a QAT prepared graph, first giving BN’s running stats and the fake quant ranges real values:
model = ConvBnReLU(3, 8).train()
with torch.no_grad():
for _ in range(5):
model(torch.randn(16, 3, 16, 16) * 2 + 1)
quantizer, _ = get_xnnpack_quantizer(is_per_channel=True, is_qat=True)
prepared = prepare_qat_pt2e(export_module(copy.deepcopy(model), example_inputs(), train=True), quantizer)
with torch.no_grad():
for _ in range(3):
prepared(torch.randn(16, 3, 16, 16) * 2 + 1)
def state(gm, name):
"""A parameter or buffer on the prepared graph, e.g. conv.weight, bn.running_var."""
params = dict(gm.named_parameters())
return params[name] if name in params else gm.get_buffer(name)
def fake_quants(gm):
"""Fake quants in forward order: input x, folded weight, relu output (the order seen in M04-2)."""
return [gm.get_submodule(n.target) for n in gm.graph.nodes
if n.op == "call_module" and "activation_post_process" in n.target]A. Hand-Write the Approximate Version and Compare with PT2E’s Graph
The approximate version is Eq. (1) of M04-2: \(s = \gamma / \sqrt{\sigma^2_{\text{run}} + \varepsilon}\), first run the conv with \(\mathrm{fq}(W \cdot s)\), then divide by \(s\), and finally hand it to BN using batch statistics. The hand-written version directly borrows the three fake quant modules and the parameters from the prepared graph, to make sure both sides use the same scales:
def forward_approximate(gm, x):
fq_x, fq_w, fq_out = fake_quants(gm)
w, gamma, beta = state(gm, "conv.weight"), state(gm, "bn.weight"), state(gm, "bn.bias")
rm, rv = state(gm, "bn.running_mean"), state(gm, "bn.running_var")
s = gamma / torch.sqrt(rv + EPS)
y = F.conv2d(fq_x(x), fq_w(w * s.reshape(-1, 1, 1, 1)), None, padding=1) / s.reshape(1, -1, 1, 1)
y = F.batch_norm(y, rm, rv, gamma, beta, True, 0.1, EPS) # training=True: use batch statistics
return fq_out(F.relu(y))Fake quant and BN both update their own state during forward, so each path gets its own independent copy:
x = torch.randn(16, 3, 16, 16) * 2 + 1
g_graph, g_approx, g_slow = (copy.deepcopy(prepared) for _ in range(3))
with torch.no_grad():
y_graph = g_graph(x)
y_approx = forward_approximate(g_approx, x)
diff = float((y_approx - y_graph).abs().max())
print(f"max|hand-written approximate - prepare_qat_pt2e graph| = {diff}")
assert diff == 0max|hand-written approximate - prepare_qat_pt2e graph| = 0.0
Bit-for-bit identical: PT2E’s QAT subgraph is PyTorch’s approximate version. The torchao 0.18 source says the same: the conv + BN pattern in qat_utils.py is documented as written after nniqat.ConvBn2d._forward_approximate, and prepare_qat_pt2e takes only the two parameters model and quantizer, with no option to switch to the slow version.
B. Hand-Write the Slow Version and See How Much It Differs
The slow version’s weight fake quant still quantizes \(\gamma W / \sigma\) (\(\sigma = \sqrt{\sigma^2_{\text{run}} + \varepsilon}\), the same as the approximate version’s \(W \cdot s\)); the difference is in the batch statistics: first run an extra conv with the unquantized \(x\) and \(W\) to get this batch’s \(\mu_B\), \(\sigma_B\), then correct the quantized conv output back to the batch statistics:
\[ y = \mathrm{conv}\big(\mathrm{fq}(x),\ \mathrm{fq}(\gamma W / \sigma)\big) \cdot \frac{\sigma}{\sigma_B} + \Big(\beta - \frac{\gamma\,\mu_B}{\sigma_B}\Big) \tag{1} \]
def forward_slow(gm, x):
fq_x, fq_w, fq_out = fake_quants(gm)
w, gamma, beta = state(gm, "conv.weight"), state(gm, "bn.weight"), state(gm, "bn.bias")
sigma = torch.sqrt(state(gm, "bn.running_var") + EPS)
y_float = F.conv2d(x, w, None, padding=1) # the extra conv: unquantized
mu_b = y_float.mean((0, 2, 3))
sigma_b = torch.sqrt(y_float.var((0, 2, 3), unbiased=False) + EPS)
w_q = fq_w(w * (gamma / sigma).reshape(-1, 1, 1, 1))
y = F.conv2d(fq_x(x), w_q, None, padding=1) * (sigma / sigma_b).reshape(1, -1, 1, 1)
y = y + (beta - gamma * mu_b / sigma_b).reshape(1, -1, 1, 1)
return fq_out(F.relu(y))
with torch.no_grad():
y_slow = forward_slow(g_slow, x)
y_fp32 = copy.deepcopy(model).train()(x) # the original model, unquantized, in train mode
results = {"approximate (= PT2E)": sqnr(y_fp32, y_approx), "slow": sqnr(y_fp32, y_slow)}
table([[k, f"{v:.2f}"] for k, v in results.items()] + [["slow vs approximate (compared with each other)", f"{sqnr(y_graph, y_slow):.2f}"]],
["", "SQNR (dB) vs the fp32 train-mode output"])| SQNR (dB) vs the fp32 train-mode output | |
|---|---|
| approximate (= PT2E) | 39.11 |
| slow | 38.89 |
| slow vs approximate (compared with each other) | 44.55 |
gap = abs(results["approximate (= PT2E)"] - results["slow"])
print(f"SQNR gap of the two versions vs fp32: {gap:.2f} dB")
assert gap < 1.0SQNR gap of the two versions vs fp32: 0.22 dB
C. Where the Difference Lies
- The fake quant inputs are exactly the same: the slow version’s \(\gamma W / \sigma\) is the approximate version’s \(W \cdot s\). The only remaining difference is where BN’s \(\mu_B\), \(\sigma_B\) come from: the slow version takes them from the unquantized conv, the approximate version from the quantized conv (M04-2 Eq. (2)).
- So the approximate version saves one conv; the cost is that quantization error mixes into the batch statistics (as the post puts it).
- In this 8-bit, single-seed measurement, the two versions’ SQNR against fp32 differ by less than 1 dB (numbers in the table above), at the noise scale, so we cannot say from this which version is more accurate.
- The post mentions that at lower bit widths (e.g. below 4 bit) the gap may no longer be negligible; only 8 bit is tested here, and the low-bit case is not verified.
- The eager QAT module can switch to the slow version (
_enable_slow_path_for_better_numerical_stability, off by default); PT2E has no such switch (section A).
Summary
- PT2E’s QAT conv + BN subgraph is bit-for-bit identical to PyTorch’s
_forward_approximate, not the whitepaper’s slow version. - The two versions differ only in whether BN’s batch statistics are taken before or after quantization; at 8 bit the gap between them is small.