import copy
import torch
from common.data import calibration_batches, evaluate, make_loader, trained_tiny_net
from common.graph_utils import sqnr
from common.models import example_inputs
from common.pt2e import convert_pt2e, export_module, get_xnnpack_quantizer, prepare_pt2e, quiet
from common.viz import table
quiet()
_ = torch.manual_seed(0)M05-1 | Full PTQ: First Decide How to Measure How Much Quantization Lost
The four PTQ steps (export, prepare, calibration, convert) were covered in M01-1. This chapter answers: when the quantized result is unsatisfactory, where do I look? Before looking, you need a ruler that is sensitive enough. This notebook compares two rulers on the same trained model:
- A. Setup: the trained model and a
ptq()function, shared by the next few notebooks. - B. Accuracy vs output SQNR: which one shows the quantization error.
- C. Comparing a setting with SQNR: per-channel vs per-tensor weights.
A. Setup
trained_tiny_net() is a tool of this project (common/data.py): with a fixed seed, it trains TinyConvNet for 20 epochs on synthetic 4-class images and returns the model in eval mode. TinyConvNet has conv + BN + relu, a residual block, pooling, and a final linear (fc); all numbers in this chapter come from it.
ptq() wraps the four steps of M01-1 into one function. Calibration data comes from calibration_batches(n, seed): synthetic data with inputs only and no labels; a different seed gives a different set.
def ptq(model, per_channel=True, calib_samples=128, seed=123):
"""export → prepare_pt2e → calibration → convert_pt2e; returns the graph after convert."""
quantizer, _ = get_xnnpack_quantizer(is_per_channel=per_channel)
prepared = prepare_pt2e(export_module(model, example_inputs()), quantizer)
with torch.no_grad():
for batch in calibration_batches(calib_samples, seed=seed):
prepared(*batch)
return convert_pt2e(copy.deepcopy(prepared))
model = trained_tiny_net()
converted = ptq(model)
test_loader = make_loader(n=1024, seed=2, shuffle=False) # test data: 1024 samples, with labels
(x_eval,) = next(calibration_batches(64, seed=9)) # a separate batch of inputs for computing SQNR, not calibration dataB. Accuracy vs Output SQNR
Output SQNR (signal-to-quantization-noise ratio) converts the difference between the quantized output and the fp32 output into dB (see the chapter overview for the definition). It only compares the outputs of two models and needs no labels. common.graph_utils.sqnr(ref, test) is this project’s function for it.
acc_fp32, acc_int8 = evaluate(model, test_loader), evaluate(converted, test_loader)
with torch.no_grad():
out_sqnr = sqnr(model(x_eval), converted(x_eval))
table([["fp32", f"{acc_fp32:.4f}", "—"], ["int8 (PTQ)", f"{acc_int8:.4f}", f"{out_sqnr:.2f}"]],
["model", "accuracy (1024 samples)", "output SQNR (dB)"])| model | accuracy (1024 samples) | output SQNR (dB) |
|---|---|---|
| fp32 | 0.7109 | — |
| int8 (PTQ) | 0.7051 | 27.32 |
print(f"accuracy gap: {abs(acc_fp32 - acc_int8):.4f}")
assert abs(acc_fp32 - acc_int8) < 0.02 and out_sqnr < 35accuracy gap: 0.0059
- Accuracy barely moved: this synthetic task is insensitive to quantization, and out of 1024 samples only a few answers differ before and after quantization. Using it to compare settings, the differences get buried in noise.
- Output SQNR is a continuous number: it shows whether quantization error grows or shrinks, and it needs no labels, so it is usable in diagnosis too.
So the following notebooks use output SQNR as the main metric and accuracy as a supplement. Whether accuracy would be more sensitive on a real task is not verified here.
C. Comparing a Setting with SQNR: per-channel vs per-tensor
Same model, same calibration data, only the weight quantization granularity changes:
rows, scores = [], {}
for name, per_channel in (("per-channel weight", True), ("per-tensor weight", False)):
c = ptq(model, per_channel=per_channel)
with torch.no_grad():
scores[name] = sqnr(model(x_eval), c(x_eval))
rows.append([name, f"{evaluate(c, test_loader):.4f}", f"{scores[name]:.2f}"])
table(rows, ["setting", "accuracy", "output SQNR (dB)"])| setting | accuracy | output SQNR (dB) |
|---|---|---|
| per-channel weight | 0.7051 | 27.32 |
| per-tensor weight | 0.7090 | 25.62 |
gap = scores["per-channel weight"] - scores["per-tensor weight"]
print(f"per-channel is {gap:.2f} dB higher than per-tensor")
assert gap > 0per-channel is 1.70 dB higher than per-tensor
- In SQNR, per-channel is better; the accuracy difference is at the noise level, so it can’t tell which is better.
- The expected benefit of per-channel: after BN is folded into conv, the value ranges of the output channels can differ a lot (M04-1 C), and per-tensor gets dragged down by the channel with the largest range.
- This is the result for one model and one calibration set; it cannot be generalized to “per-channel is always several dB better”. The effect of a setting has to be measured with data.
Summary
- On this task accuracy is insensitive to quantization; output SQNR is continuous and needs no labels, so it is this chapter’s main metric.
- With a ruler in hand, settings can be compared: per-channel has a higher SQNR on this model.
- The next notebook, M05-2, uses the same ruler to look at the effect of calibration data.