Skip to main content

M06 Overview | The QAT Training Pipeline: Make the Training Graph Equal the Graph After convert

Open In Colab

M05 used PTQ to quantize a trained model and learned how to track down accuracy problems. When the bit width goes lower, PTQ may really not be enough. Then we can let the model see the quantization error during training and adapt to it by itself. That is QAT (quantization-aware training). We saw what a QAT graph looks like in M04-2; this chapter is only about training. The six notebooks that follow (M06-1 to M06-6; M06-5 and M06-6 are optional) verify each point hands-on.

Questions This Chapter Answers

Question One-line answer Verified in
Can the prepared QAT graph be trained directly? Yes. It is an ordinary GraphModule; the gradient of round comes from STE M06-1
What to do before evaluation? Switch BN to eval and turn off the observers, otherwise results are not reproducible M06-2
Is the training graph equal to the graph after convert? Not always: XNNPACK’s default weight fake quant makes them unequal M06-3
At low bit widths, is QAT really better than PTQ? Clearly better at int4; running PTQ after retraining the fp32 model with the same recipe gives no improvement, so the gain comes from quantization adaptation M06-4

Two optional notebooks: M06-5 lets gradients learn the scale too (range learning); M06-6 handles QAT when the weight is computed from other parameters (weight_norm). See the two “Optional” sections below.

An Example First: QAT Wins Back the Accuracy PTQ Lost at int4

The model is the same as in M05 (the trained TinyConvNet). At int8, PTQ is already right next to fp32, so there is nothing to rescue (M06-1). So here we use int4: activation [-8, 7], weight [-7, 7]. We compare the test loss of PTQ int4 and QAT int4 (trained for 5 epochs), and check whether the trained QAT graph and its post-convert graph give the same output:

Model test loss
fp32 0.826
PTQ int4 1.064
QAT int4 0.959
QAT won back 44% of the extra test loss that PTQ int4 has over fp32
Trained QAT fake quant graph vs the graph after convert: SQNR inf dB, max difference 0.00 output LSB
  • The test loss of PTQ int4 is a good bit higher than fp32; QAT wins back part of it after only 5 epochs (the ratio is printed above). This is just one seed; the comparison over 5 seeds and the control group is in M06-4.
  • The last line is the most important check of this chapter: the graph the model sees during training (the fake quant graph) and the Q/DQ graph after convert give the same output: SQNR is inf, or the max difference is within 1 output LSB (for the criterion, see “The Principle of the Whole Chapter” below). This does not hold automatically; M06-3 and M06-5 each catch one place where it breaks.

qat_quantizer, train_qat, and freeze_for_eval used here are all this project’s tools (common/qat.py), not torchao APIs. M06-1 writes the training loop out line by line, and only then switches to these tools.

Definitions

fake quant

What QAT inserts into the graph is not an observer but a fake quant: in forward it quantizes and immediately dequantizes, so the output is still fp32 but the values already sit on the quantization grid. With scale \(s\), zero point \(z\), and integer range \([q_{min}, q_{max}]\):

\[ \hat{x} = \big(\operatorname{clamp}(\operatorname{round}(x/s) + z,\ q_{min},\ q_{max}) - z\big)\cdot s \tag{1} \]

STE (straight-through estimator)

The gradient of round is 0 almost everywhere, so differentiating directly cannot train anything. In backward, STE treats round as the identity function and keeps only the effect of clamp:

\[ \frac{\partial \hat{x}}{\partial x} = \begin{cases} 1 & q_{min} \le \operatorname{round}(x/s) + z \le q_{max} \\ 0 & \text{otherwise} \end{cases} \tag{2} \]

Compare an int4 FakeQuantize against Eq. (1) and (2):

fq = FakeQuantize(observer=MovingAverageMinMaxObserver, dtype=torch.int8, qscheme=torch.per_tensor_affine,
                  quant_min=-8, quant_max=7, eps=2**-12)
with torch.no_grad():
    fq(torch.linspace(-1, 1, 100))  # calibrate the range
fq.apply(disable_observer)
s, z = (float(v) for v in fq.calculate_qparams())

x = torch.linspace(-2, 2, 41, requires_grad=True)
y = fq(x)
y.sum().backward()
q = torch.round(x.detach() / s) + z
by_eq1 = (q.clamp(-8, 7) - z) * s                       # Eq. (1)
by_eq2 = ((q >= -8) & (q <= 7)).float()                 # Eq. (2)
assert x.grad is not None
print(f"s = {s:.4f}, z = {z:.0f}; matches Eq. (1): {torch.allclose(y, by_eq1)}; matches Eq. (2): {torch.equal(x.grad, by_eq2)}")
assert torch.allclose(y, by_eq1) and torch.equal(x.grad, by_eq2)
s = 0.1333, z = -1; matches Eq. (1): True; matches Eq. (2): True

Values outside the range get gradient 0, so this step gives them no update. The derivation of STE is in the blog post “Understanding Fake Quantization in Quantization Aware Training” (in Chinese); improvements to STE (EWGS, MAD, etc.) are collected in the QAT section of the blog post “Quantization technique roadmap” (in Chinese).

Metric: test loss, compared against “fp32 retrained with the same recipe”

M05 used “SQNR against the fp32 output”. QAT changes the weights, so that SQNR measures “how far from the original model”, not “how good after quantization”. So this chapter switches to test loss (cross-entropy: continuous, and the quantity QAT optimizes directly). QAT adds one more thing, “training”. To separate the effect of training itself from that of quantization, the baseline is the fp32 retrained with the same recipe without quantization (M06-1 C); at int4 we also add the control “PTQ after retraining” (M06-4). Merely changing the data order via the seed already changes the test loss, so M06-4 runs 5 seeds per group.

Flow

export(train=True)        BN must be the train version of the op
  → prepare_qat_pt2e      insert fake quant, build the conv + BN QAT subgraph (M04-2)
  → training              switch train/eval every epoch (M06-1)
  → freeze_for_eval       BN uses running stats, observers off (M06-2)
  → convert_pt2e          get the Q/DQ graph

The Principle of the Whole Chapter: Training Graph = Graph After convert

The premise of QAT is “the quantization error simulated during training = the quantization error at deployment”. The check is fixed: feed the same data to the fake quant graph (after freeze) and the convert graph, and compare the outputs. This chapter catches two places where the equality fails:

Where Why the equality fails Fix
M06-3 The weight’s fused fake quant computes scale with its own formula Use the non-fused FakeQuantize for weight
M06-5 The learnable scale is pushed negative by gradients, and convert clamps it to eps Clamp scale back to positive after every step

The criterion is: output SQNR is inf, or the max difference is within 1 output LSB (the scale of the output dequantize). The latter is because, for the activation’s fused fake quant, the scale occasionally differs from the value used by convert by 1 ULP, which is just floating-point rounding (M06-3 C).

This check only guarantees “training graph = Q/DQ graph”; the part from the Q/DQ graph to the backend’s integer kernels is covered from M07 on.

Optional 1: Let Gradients Learn the Scale Too (M06-5)

Up to M06-4, scale was decided by the observer’s min/max statistics, and training changed only the weights. LearnableFakeQuantize makes scale and zero point nn.Parameters so that the loss gradient adjusts the quantization range (the LSQ approach). Plugging it into PT2E only requires swapping the fake quant class in the spec, but two things need handling yourself:

  • The scale is a parameter created only on the first forward, so the optimizer must be built after that.
  • torchao only clamps scale to positive at convert time; during training the gradient may push it negative, and then the training graph no longer equals the convert graph, so clamp it back to positive at every step.

Range learning lowered the mean test loss relative to the observer version on both platforms we tried; the price is a new set of hyperparameters (grad scaling, the lr for qparams), and a wrong choice diverges. On this small int4 model it won on all 3 seeds on macOS arm64 but not on every seed on Linux x86 (see M06-5 D). The derivation of the gradients is in the blog post “Learning Zero-Point and Scale in Quantization Parameters”.

Optional 2: When the Weight Is Computed from Other Parameters (M06-6)

The “Optional” section of the M05 overview introduced torch.nn.utils.parametrize: once registered, a parameter becomes “computed from other parameters” and is recomputed on every access (PyTorch tutorial Parametrizations). The example is weight_norm, computed per output channel:

\[ W = g \cdot \frac{v}{\lVert v \rVert} \tag{3} \]

After export, Eq. (3) becomes aten._weight_norm in the graph, and the conv’s weight is its output, not a get_attr. PTQ does not train, so replacing it with a constant before prepare is enough; QAT is different, because what gets trained is exactly \(g\) and \(v\).

Where the fake quant goes and how gradients flow

Like the observer in PTQ, the weight’s fake quant is attached to the output of _weight_norm, so it quantizes the computed \(W\). In backward, the STE gradient passes through the fake quant and then flows back to \(g\) and \(v\) through _weight_norm. So weight_norm stays in effect during QAT, and what gets trained is \(g\) and \(v\).

With BN, which step fails

The torchao internal function that reads constants accepts only get_attr, so the step “read the conv’s weight out as a constant” fails:

ValueError: Expected get_attr node, got call_function

PTQ and QAT hit it at different steps:

Flow When BN is folded Result
PTQ At the start of prepare_pt2e (M04-1) Fails already in prepare
QAT prepare_qat_pt2e only builds the QAT subgraph and does not fold (M04-2); convert_pt2e folds Prepare passes and training runs as usual; the error appears only at convert, after all training is done

How to fix it

Compute \(W\) and store it as an ordinary parameter, and BN can be folded as usual. The key is when you do it:

When What to do Result
Before export remove_parametrizations(..., leave_parametrized=True) Equivalent to giving up training \(g\) and \(v\)
After training, before convert strip_weight_norm(gm): replace _weight_norm in the graph with a constant \(g\) and \(v\) were trained, and convert can fold BN

strip_weight_norm is this project’s tool (common/weight_norm.py), not a torch or torchao API; M05-5 already introduced it. Readers only need to know what it does; that the output is bitwise unchanged after the replacement and that the convert graph matches the fake quant graph are both verified in section C of M06-6.

So the QAT flow that has BN and also keeps weight_norm is: export(train=True) → prepare_qat_pt2e → training → freeze_for_eval → strip_weight_norm → convert_pt2e.

Summary and Chapter Map

  • A QAT graph can be trained directly, with the round gradient coming from STE; before evaluation and convert, switch BN to eval and turn off the observers.
  • The principle of the whole chapter is “training graph = graph after convert”, checked by comparing the outputs of the two graphs.
  • At int4, QAT is clearly better than PTQ; running PTQ after retraining fp32 with the same recipe gives no improvement, so the gain comes from simulating quantization during training, not from extra training.
  • (Optional) Range learning adds hyperparameters to tune, and its edge over the observer version holds on the mean but not on every seed and platform; weight_norm must be kept until training ends and removed only before convert.
notebook Question it answers
M06-1 Minimal QAT Gradient of STE; the minimal training loop; same shape as PTQ after convert
M06-2 Two switches before evaluation What happens if BN and observer are not switched; allow_exported_model_train_eval
M06-3 The trap of fused fake quant Why the training graph ≠ convert graph under XNNPACK’s default config; the scale formula
M06-4 Controlled experiment at int4 Whether QAT beats PTQ at low bit widths; whether the gain comes from quantization adaptation or extra training
M06-5 (optional) range learning How to plug in LearnableFakeQuantize, what to watch for, the effect
M06-6 (optional) weight_norm Keep training \(g\) and \(v\); the convert error with BN and the fix

What this chapter deliberately skips: what the Q/DQ graph gets rewritten into later, and how far it is from the backend’s integer arithmetic (from M07 on).

References