Skip to main content

M05-1 | Full PTQ: First Decide How to Measure How Much Quantization Lost

Open In Colab

The four PTQ steps (export, prepare, calibration, convert) were covered in M01-1. This chapter answers: when the quantized result is unsatisfactory, where do I look? Before looking, you need a ruler that is sensitive enough. This notebook compares two rulers on the same trained model:

import copy

import torch

from common.data import calibration_batches, evaluate, make_loader, trained_tiny_net
from common.graph_utils import sqnr
from common.models import example_inputs
from common.pt2e import convert_pt2e, export_module, get_xnnpack_quantizer, prepare_pt2e, quiet
from common.viz import table

quiet()
_ = torch.manual_seed(0)

A. Setup

trained_tiny_net() is a tool of this project (common/data.py): with a fixed seed, it trains TinyConvNet for 20 epochs on synthetic 4-class images and returns the model in eval mode. TinyConvNet has conv + BN + relu, a residual block, pooling, and a final linear (fc); all numbers in this chapter come from it.

ptq() wraps the four steps of M01-1 into one function. Calibration data comes from calibration_batches(n, seed): synthetic data with inputs only and no labels; a different seed gives a different set.

def ptq(model, per_channel=True, calib_samples=128, seed=123):
    """export → prepare_pt2e → calibration → convert_pt2e; returns the graph after convert."""
    quantizer, _ = get_xnnpack_quantizer(is_per_channel=per_channel)
    prepared = prepare_pt2e(export_module(model, example_inputs()), quantizer)
    with torch.no_grad():
        for batch in calibration_batches(calib_samples, seed=seed):
            prepared(*batch)
    return convert_pt2e(copy.deepcopy(prepared))


model = trained_tiny_net()
converted = ptq(model)
test_loader = make_loader(n=1024, seed=2, shuffle=False)  # test data: 1024 samples, with labels
(x_eval,) = next(calibration_batches(64, seed=9))  # a separate batch of inputs for computing SQNR, not calibration data

B. Accuracy vs Output SQNR

Output SQNR (signal-to-quantization-noise ratio) converts the difference between the quantized output and the fp32 output into dB (see the chapter overview for the definition). It only compares the outputs of two models and needs no labels. common.graph_utils.sqnr(ref, test) is this project’s function for it.

acc_fp32, acc_int8 = evaluate(model, test_loader), evaluate(converted, test_loader)
with torch.no_grad():
    out_sqnr = sqnr(model(x_eval), converted(x_eval))
table([["fp32", f"{acc_fp32:.4f}", "—"], ["int8 (PTQ)", f"{acc_int8:.4f}", f"{out_sqnr:.2f}"]],
      ["model", "accuracy (1024 samples)", "output SQNR (dB)"])
model accuracy (1024 samples) output SQNR (dB)
fp32 0.7109 —
int8 (PTQ) 0.7051 27.32
print(f"accuracy gap: {abs(acc_fp32 - acc_int8):.4f}")
assert abs(acc_fp32 - acc_int8) < 0.02 and out_sqnr < 35
accuracy gap: 0.0059
  • Accuracy barely moved: this synthetic task is insensitive to quantization, and out of 1024 samples only a few answers differ before and after quantization. Using it to compare settings, the differences get buried in noise.
  • Output SQNR is a continuous number: it shows whether quantization error grows or shrinks, and it needs no labels, so it is usable in diagnosis too.

So the following notebooks use output SQNR as the main metric and accuracy as a supplement. Whether accuracy would be more sensitive on a real task is not verified here.

C. Comparing a Setting with SQNR: per-channel vs per-tensor

Same model, same calibration data, only the weight quantization granularity changes:

rows, scores = [], {}
for name, per_channel in (("per-channel weight", True), ("per-tensor weight", False)):
    c = ptq(model, per_channel=per_channel)
    with torch.no_grad():
        scores[name] = sqnr(model(x_eval), c(x_eval))
    rows.append([name, f"{evaluate(c, test_loader):.4f}", f"{scores[name]:.2f}"])
table(rows, ["setting", "accuracy", "output SQNR (dB)"])
setting accuracy output SQNR (dB)
per-channel weight 0.7051 27.32
per-tensor weight 0.7090 25.62
gap = scores["per-channel weight"] - scores["per-tensor weight"]
print(f"per-channel is {gap:.2f} dB higher than per-tensor")
assert gap > 0
per-channel is 1.70 dB higher than per-tensor
  • In SQNR, per-channel is better; the accuracy difference is at the noise level, so it can’t tell which is better.
  • The expected benefit of per-channel: after BN is folded into conv, the value ranges of the output channels can differ a lot (M04-1 C), and per-tensor gets dragged down by the channel with the largest range.
  • This is the result for one model and one calibration set; it cannot be generalized to “per-channel is always several dB better”. The effect of a setting has to be measured with data.

Summary

  • On this task accuracy is insensitive to quantization; output SQNR is continuous and needs no labels, so it is this chapter’s main metric.
  • With a ruler in hand, settings can be compared: per-channel has a higher SQNR on this model.
  • The next notebook, M05-2, uses the same ruler to look at the effect of calibration data.