| model | accuracy (1024 samples) | output SQNR (dB) |
|---|---|---|
| fp32 | 0.7109 | — |
| int8 (PTQ) | 0.7051 | 27.32 |
M05 Overview | Debugging PTQ Accuracy: Measure First, Narrow Down, Then Confirm by Experiment
M01 walked through the four steps of PTQ. When you apply it to a real model, the most common problem is: accuracy dropped after quantization, where do I look? This chapter uses one trained model to build an order of investigation. The five notebooks that follow (M05-1 to M05-5; M05-5 is optional) verify each step hands-on.
Why We Need an Order of Investigation
Quantization error can come from many places: too little or unrepresentative calibration data, a layer whose value range is especially hard to quantize, an output that is especially sensitive to error… Guessing and changing settings at random often means hours of tweaking with no progress. This chapter answers four questions:
| Question | One-line answer | Verify hands-on |
|---|---|---|
| How do we measure how much quantization lost? | Use output SQNR; accuracy is often not sensitive enough | M05-1 |
| Is there enough calibration data? | Rerun with a few different calibration sets and look at the spread of results | M05-2 |
| Where does the error come from? | The numeric debugger gives every location an SQNR, used to narrow things down | M05-3 |
| Can part of the model stay in fp32? | Yes; but confirm by experiment which exclusion actually helps | M05-4 |
The optional M05-5 is a different topic: how PTQ handles a weight that is not a stored parameter but is “computed from other parameters” (see the “Optional” section below).
An Example First: Accuracy Barely Changes, but SQNR Is Only 27 dB
The model in this chapter is a trained TinyConvNet (conv + BN + relu, a residual block, pooling, and a final linear), trained with a fixed seed by this project’s trained_tiny_net() (common/data.py). Do one PTQ with XNNPACKQuantizer and compare before and after quantization:
Accuracy differs by only a fraction of a percentage point, so quantization looks nearly lossless; but the output SQNR tells us there is a certain amount of error between the quantized output and fp32. This synthetic task is insensitive to quantization and accuracy gets buried in noise, so this chapter uses SQNR as the main metric.
Definitions
Output SQNR
For the same batch of inputs, let the fp32 model’s output be \(y\) and the quantized output be \(\hat{y}\). SQNR (signal-to-quantization-noise ratio) is the ratio of signal power to error power, expressed in dB:
\[ \mathrm{SQNR} = 10 \log_{10} \frac{\operatorname{mean}(y^2)}{\operatorname{mean}\big((y - \hat{y})^2\big)} \quad \text{(dB)} \tag{1} \]
Larger means closer to fp32. Put another way: every 20 dB more SQNR makes the RMS of the error 10 times smaller. 27 dB means the RMS of the error is about 4.5% of the signal. It only compares the outputs of two models and needs no labels. This project computes it with common.graph_utils.sqnr(ref, test); check it against equation (1):
y = torch.randn(1000)
y_hat = y + 0.05 * torch.randn(1000)
by_formula = float(10 * torch.log10(y.pow(2).mean() / (y - y_hat).pow(2).mean())) # Eq. (1)
print(f"Eq. (1): {by_formula:.3f} dB; sqnr(): {sqnr(y, y_hat):.3f} dB")
assert abs(by_formula - sqnr(y, y_hat)) < 1e-4Eq. (1): 26.263 dB; sqnr(): 26.263 dB
Layerwise SQNR
Equation (1) can also be computed on the output of every node in the middle of the graph: run the reference graph and the quantized graph on the same input and compare at the same node. torchao’s numeric debugger does this (M05-3). The reference graph should be a copy taken “after prepare_pt2e, before calibration”: BN is already folded and the ops match the quantized graph one to one, so the only difference is quantization.
Mixed precision
Keep part of the model in fp32 and quantize the rest. The way to do it is to have the Quantizer not annotate that part (M03-6); M05-4 writes a SkipQuantizer that wraps XNNPACKQuantizer to do this.
The Order of Investigation
overall output SQNR Is there a problem? (M05-1)
→ a few calibration sets Are the results stable? If the spread is large, add calibration data first (M05-2)
→ layerwise SQNR Where does the error grow? Find candidates (M05-3)
→ exclude a candidate, remeasure Does the overall SQNR really recover? (M05-4)
→ pick the least invasive fix e.g. only drop the output quantization of one module (M05-4 C)
This chapter walks through it on this model: the layerwise SQNR drops most at head_conv, but keeping it in fp32 does not help; what actually helps is the final fc, and removing just its output quantization is enough. Layerwise scores are clues, not conclusions.
Optional: When the Weight Is Computed from Other Parameters (M05-5)
torch.nn.utils.parametrize lets a parameter become “computed from other parameters”: once registered, every access to conv.weight recomputes it. For usage see the PyTorch tutorial Parametrizations and the docs of register_parametrization.
A common example is weight_norm. It splits the weight into a direction \(v\) and a magnitude \(g\), computed separately for each output channel:
\[ W = g \cdot \frac{v}{\lVert v \rVert} \tag{2} \]
During training the parameters being updated are \(g\) and \(v\); \(W\) is just a computed value.
Where the Observer Goes
When export traces forward, it records equation (2) into the graph as well: the conv’s weight is no longer a get_attr node reading a parameter, but the output of the aten._weight_norm op. The Quantizer annotates the conv’s weight input, so the observer attaches to the output of _weight_norm. What is quantized is the computed \(W\); \(g\) and \(v\) themselves are not quantized.
This is exactly the place that should be quantized: at inference the conv only uses \(W\). Without BN, prepare and convert both work fine; convert defaults to fold_quantize=True, which precomputes _weight_norm together with the weight’s quantize into an int8 constant, so \(g\) and \(v\) no longer appear in the graph.
With BN, prepare_pt2e Raises an Error
prepare_pt2e folds BN into conv right at the start (M04-1): it reads the conv’s weight as a constant, multiplies by BN’s coefficients, and writes it back. The weight is the output of _weight_norm, not a constant, so it errors out directly instead of silently skipping:
ValueError: Expected get_attr node, got call_function
How to Fix It
PTQ does not train and has no use for \(g\) and \(v\). Compute \(W\), store it as a regular parameter, and BN can be folded as usual. There are two ways:
| When | How | Watch out for |
|---|---|---|
| Before export | remove_parametrizations(..., leave_parametrized=True): store the currently computed weight as a regular parameter |
Call it only on a deepcopy’d clone, or the original model breaks too |
| After export | strip_weight_norm(gm): replace the _weight_norm in the graph with a constant |
Only handles weight_norm |
strip_weight_norm is a tool of this project (common/weight_norm.py), not a torch or torchao API. The reader only needs to know what it does; how it edits the graph, and why the deepcopy trap happens, are both in section D of M05-5.
QAT has to update \(g\) and \(v\) during training, so they cannot be removed before prepare and the approach is different; see M06-6.
Summary and Chapter Map
- Decide how to measure first: output SQNR is continuous and needs no labels, far more sensitive than accuracy on this task.
- Look at the spread across calibration data; use layerwise SQNR to narrow down; finally confirm by experiment and choose the least invasive change.
- (Optional) When the weight is computed by parametrize, the observer quantizes the computed weight; with BN, replace it with a constant first.
| notebook | Question it answers |
|---|---|
| M05-1 Full PTQ and How to Measure | Which of accuracy and SQNR shows the quantization error; per-channel vs per-tensor |
| M05-2 Calibration | Number of calibration samples and the spread of results; whether BN affects results during calibration |
| M05-3 Layerwise Diagnosis | How to use and read the numeric debugger, and which reference graph to use |
| M05-4 Mixed precision | Which module’s exclusion really helps, and how error accumulates |
| M05-5 (Optional) weight_norm | A weight computed from other parameters: where the observer goes, why BN causes an error, two fixes |
What this chapter deliberately skips: recovering with training when PTQ is really not enough (QAT, M06); how far the backend’s actual integer arithmetic is from the Q/DQ graph (M07 onward); improving the PTQ algorithm itself (e.g. AdaRound, OCTAV), see the PTQ section of the blog post “Quantization Roadmap” (in Chinese).
References
- Official PyTorch tutorial: PyTorch 2 Export Post Training Quantization
- torchao 0.18 source:
_numeric_debugger.py(numeric debugger) - R. Krishnamoorthi, Quantizing deep convolutional networks for efficient inference: A whitepaper, 2018: experiments on per-channel, calibration, and quantization error
- PyTorch tutorial: Parametrizations; docs:
register_parametrization,weight_norm(used by optional M05-5) - Notes on Weight Normalization (in Chinese): why weight_norm splits the weight into \(g\) and \(v\), and the gradients of both (used by optional M05-5)