import copy
import torch
from common.data import calibration_batches, trained_tiny_net
from common.graph_utils import sqnr
from common.models import example_inputs
from common.pt2e import (
FakeQuantize,
FusedMovingAvgObsFakeQuantize,
MovingAverageMinMaxObserver,
MovingAveragePerChannelMinMaxObserver,
convert_pt2e,
disable_observer,
export_module,
prepare_qat_pt2e,
quiet,
)
from common.qat import EPS, freeze_for_eval, qat_quantizer, train_qat
from common.viz import table
quiet()
_ = torch.manual_seed(0)M06-3 | Training Graph ≠ Graph After convert? A Trap in XNNPACK’s Default QAT Config
The premise of QAT is “the quantization error simulated during training = the quantization error at inference”. This notebook checks that premise: feed the same data to the trained QAT graph (the fake quant graph, eval mode, observers off) and to its post-convert Q/DQ graph, and the outputs should be exactly the same.
What fused fake quant means: this “fused” has nothing to do with M04’s conv + BN folding. A FakeQuantize module does three steps internally:
- The observer updates min/max.
- Compute scale/zero_point from min/max.
- quantize → clamp → dequantize.
FusedMovingAvgObsFakeQuantize merges the three steps into one ATen kernel (torch.fused_moving_avg_obs_fake_quant) that does them all at once, and XNNPACKQuantizer’s QAT config (is_qat=True) uses it by default. The cost is that step 2 is computed by this kernel itself, without going through the torchao observer’s calculate_qparams(), and that is where the problem comes from.
- A. Check: with the weight using XNNPACK’s default fused fake quant, how far apart the two graphs are at int8/int4.
- B. Cause: the formula the fused kernel uses for scale differs from
calculate_qparams()(which convert uses). - C. Fix: use the non-fused
FakeQuantizefor weight.
A. Fake Quant Graph vs Graph After convert
qat_quantizer(weight_fq=...) (common/qat.py) lets you choose the weight’s fake quant: "fused" is XNNPACK’s default FusedMovingAvgObsFakeQuantize, and "plain" is the non-fused FakeQuantize. The activation uses fused in both. Each config trains for 2 epochs, then freeze_for_eval (M06-2) and convert, and we compare the output SQNR of the two graphs on the same input:
model = trained_tiny_net()
(x,) = next(calibration_batches(64, seed=9))
def fq_vs_converted(bits, weight_fq):
prepared = prepare_qat_pt2e(export_module(copy.deepcopy(model), example_inputs(), train=True),
qat_quantizer(bits=bits, weight_fq=weight_fq))
freeze_for_eval(train_qat(prepared, epochs=2))
converted = convert_pt2e(copy.deepcopy(prepared))
with torch.no_grad():
return sqnr(prepared(x), converted(x))
a = {(wfq, bits): fq_vs_converted(bits, wfq) for wfq in ("fused", "plain") for bits in (8, 4)}
table([["FusedMovingAvgObsFakeQuantize (XNNPACK default)", f"{a['fused', 8]:.1f}", f"{a['fused', 4]:.1f}"],
["FakeQuantize (non-fused)", f"{a['plain', 8]:.1f}", f"{a['plain', 4]:.1f}"]],
["Fake quant for weight", "int8 (dB)", "int4 (dB)"])| Fake quant for weight | int8 (dB) | int4 (dB) |
|---|---|---|
| FusedMovingAvgObsFakeQuantize (XNNPACK default) | 30.6 | 4.3 |
| FakeQuantize (non-fused) | inf | inf |
assert a["fused", 8] < 40 and a["fused", 4] < a["fused", 8]
assert a["plain", 8] == float("inf") and a["plain", 4] == float("inf")- Under the default config the two graphs are different: int8 is only around 30 dB, and int4 drops to single-digit dB. The quantization grid the model adapts to during training is not the grid of the post-convert Q/DQ graph, so QAT is not training the model that convert produces.
- The activation config is the same in both rows, so the problem is in the weight, that is, the per-channel symmetric fake quant.
B. Cause: the Scale Formula of the Fused Kernel
How each side computes scale:
- During training: the fused kernel computes it itself inside forward. In torch v2.14.0,
ChooseQuantizationParams(QuantUtils.hlines 86–93) first converts \([q_{min}, q_{max}]\) into \([-(n+1), n]\) for the symmetric case, with \(n = \lfloor (q_{max}-q_{min})/2 \rfloor\), and then computes
\[ s_{train} = \max\!\Big(\frac{|min|}{n+1}, \frac{|max|}{n}\Big) \cdot \frac{2n+1}{q_{max}-q_{min}} \tag{1} \]
- At convert time:
convert_pt2ecallscalculate_qparams(), which delegates to the torchao observer:
\[ s_{convert} = \frac{\max(|min|, |max|)}{(q_{max}-q_{min})/2} \tag{2} \]
Take the int8 weight’s \([-127, 127]\) as an example; Eq. (1) uses \([-128, 127]\). Below we run a fused fake quant once on a set of weights and compare its scale buffer (used in training) with calculate_qparams() (used by convert), and check each against Eq. (1) and (2):
def scale_eq1(mn, mx, qmin, qmax):
n = (qmax - qmin) // 2
return torch.maximum(mn.abs() / (n + 1), mx.abs() / n) * (2 * n + 1) / (qmax - qmin)
def scale_eq2(mn, mx, qmin, qmax):
return torch.maximum(-mn, mx) / ((qmax - qmin) / 2)
def fused(qmin, qmax, per_channel=True):
if per_channel:
return FusedMovingAvgObsFakeQuantize(observer=MovingAveragePerChannelMinMaxObserver, ch_axis=0, dtype=torch.int8,
qscheme=torch.per_channel_symmetric, quant_min=qmin, quant_max=qmax, eps=EPS)
return FusedMovingAvgObsFakeQuantize(observer=MovingAverageMinMaxObserver, dtype=torch.int8,
qscheme=torch.per_tensor_affine, quant_min=qmin, quant_max=qmax, eps=EPS)
w = torch.randn(6, 3, 3, 3)
w[0] *= 3 # some channels are dominated by |min|, others by max; the formulas differ in different directions
w[1, 0, 0, 0] = 5.0
rows, checks = [], []
for qmin, qmax in ((-127, 127), (-7, 7)):
fq = fused(qmin, qmax)
with torch.no_grad():
fq(w)
obs = fq.activation_post_process
s_train, s_convert = fq.scale.flatten(), fq.calculate_qparams()[0].flatten()
ok1 = torch.allclose(s_train, scale_eq1(obs.min_val, obs.max_val, qmin, qmax), rtol=1e-5)
ok2 = torch.allclose(s_convert, scale_eq2(obs.min_val, obs.max_val, qmin, qmax), rtol=1e-5)
ratio = s_train / s_convert
rows.append([f"per-channel symmetric [{qmin}, {qmax}]", f"{float(ratio.min()):.4f} ~ {float(ratio.max()):.4f}", ok1, ok2])
checks += [ok1, ok2]
fq = fused(-128, 127, per_channel=False)
with torch.no_grad():
fq(w)
rows.append(["per-tensor affine [-128, 127] (used by activation)", f"{float(fq.scale / fq.calculate_qparams()[0]):.4f}", "—", "—"])
table(rows, ["Config", "scale ratio: training / convert", "Training scale matches Eq. (1)", "Convert scale matches Eq. (2)"])| Config | scale ratio: training / convert | Training scale matches Eq. (1) | Convert scale matches Eq. (2) |
|---|---|---|---|
| per-channel symmetric [-127, 127] | 0.9961 ~ 1.0039 | True | True |
| per-channel symmetric [-7, 7] | 0.9375 ~ 1.0714 | True | True |
| per-tensor affine [-128, 127] (used by activation) | 1.0000 | — | — |
assert all(checks)int8 is off by ±0.4%, int4 by −6% to +7%. Per-tensor affine does not take this symmetric branch, so the two sides agree.
What about after disable_observer? The same version’s fused_obs_fake_quant.cpp is clear: observer_on only decides whether to update min/max; as long as fake_quant_on is set, every forward recomputes scale with Eq. (1). Verification: turn the observer off, manually set the scale buffer to the value convert uses, and forward once more:
fq = fused(-127, 127)
with torch.no_grad():
fq(w)
fq.apply(disable_observer)
s_before = fq.scale.clone()
with torch.no_grad():
fq.scale.copy_(fq.calculate_qparams()[0]) # set to the value convert uses
fq(w) # forward once more
print("After forward, scale is back to the value of Eq. (1):", torch.allclose(fq.scale, s_before))
assert torch.allclose(fq.scale, s_before)After forward, scale is back to the value of Eq. (1): True
So disable_observer cannot save you: as long as the weight’s fake quant is fused, training uses Eq. (1).
C. Fix: Use the Non-Fused FakeQuantize for Weight
FakeQuantize.forward does the three steps separately: observer → calculate_qparams() → fake_quantize_per_channel_affine. The second step uses the same formula as convert:
plain = FakeQuantize(observer=MovingAveragePerChannelMinMaxObserver, ch_axis=0, dtype=torch.int8,
qscheme=torch.per_channel_symmetric, quant_min=-7, quant_max=7, eps=EPS)
with torch.no_grad():
plain(w)
ratio = plain.scale / plain.calculate_qparams()[0]
print(f"FakeQuantize: scale used in forward ÷ calculate_qparams() = {float(ratio.min()):.4f} ~ {float(ratio.max()):.4f}")
assert torch.equal(plain.scale, plain.calculate_qparams()[0])FakeQuantize: scale used in forward ÷ calculate_qparams() = 1.0000 ~ 1.0000
The second row of the table in section A is therefore inf. qat_quantizer() in common/qat.py is configured this way by default: the activation stays fused (per-tensor affine, where the formulas agree), and the weight uses FakeQuantize. All QAT after M06 uses it.
- The cost is losing the speed of the fused kernel. The small model in this chapter would not feel it; it is not measured here.
- Why the convert graph is the one to align with: when the XNNPACK backend deploys, it reads the scale on the q/dq of the convert graph (M08), so whatever convert computes is what gets deployed.
The check is always the same from now on: the output SQNR of the fake quant graph vs the convert graph should be inf. There is one exception: for the activation’s fused fake quant, the scale buffer occasionally differs from calculate_qparams() by 1 ULP (depending on the seed). Then the max output difference is within 1 output LSB (the scale of the output dq), and SQNR may drop to around 50 dB, but this is only floating-point rounding and also counts as a pass. The roughly 30 dB and single-digit dB here come from different formulas, not from rounding.
This check only guarantees “training graph = Q/DQ graph”; the part from the Q/DQ graph to the backend’s integer kernels is measured separately in M08–M09.
Summary
- Under XNNPACK’s default QAT config, the weight uses fused fake quant, and the training graph differs from the convert graph (about 30 dB at int8, single-digit dB at int4).
- Cause: the fused kernel’s symmetric scale formula (Eq. 1) differs from the observer’s (Eq. 2), and it recomputes on every forward even after
disable_observer. - Fix: use the non-fused
FakeQuantizefor weight; the criterion is SQNR of inf, or a max difference within 1 output LSB.