Skip to main content

M10-1 | Writing the VPU’s Hardware Constraints as a Quantizer: MyVPUQuantizer

Open In Colab

The Quantizers in M03 were rules made up for teaching; a real backend Quantizer translates hardware constraints into annotations. This chapter invents a VPU, and its constraints and implementation are in vpu_quantizer.py in the same directory (the project’s backend itself, not a torchao API); read it side by side with this notebook.

import torch

from common.pt2e import MinMaxObserver, convert_pt2e, export_module, get_xnnpack_quantizer, prepare_pt2e, quiet
from common.data import calibration_batches, make_tensors, trained_tiny_net
from common.graph_utils import observer_nodes, qdq_nodes, short, sqnr
from common.models import example_inputs
from common.viz import draw, table
from vpu_quantizer import VPU_OPS, MyVPUQuantizer, Pow2MinMaxObserver

quiet()
_ = torch.manual_seed(0)
BATCH = 16

vpu_quantizer.py sits in the same folder as this notebook, so you can from vpu_quantizer import … directly.

A. The Eight Constraints → PT2E Mechanism → Where in the Code

The VPU is fictional, but every constraint is a common requirement of VPU/NPU-class hardware (general knowledge; this project has no real device):

VPU constraint PT2E approach Where in vpu_quantizer.py
(1) only VPU_OPS have kernels other ops get no annotation → stay in fp32 (M03-6) VPU_OPS, the end of annotate()
(2) activation per-tensor symmetric QuantizationSpec(qscheme=per_tensor_symmetric) act_spec()
(3) weight per-channel symmetric QuantizationSpec(qscheme=per_channel_symmetric, ch_axis=0) weight_spec()
(4) scale is a power of 2 subclass MinMaxObserver, override calculate_qparams() Pow2MinMaxObserver etc.
(5) bias int32, scale = s_in × s_w DerivedQuantizationSpec (M03-5) bias_spec()
(6) add/cat/pool etc. share one scale between input and output SharedQuantizationSpec (M03-4) _add(), _cat(), _same_scale()
(7) conv/add + relu fused put the output annotation on the relu, none in between (M04-1) fused_relu()
(8) static shape only validate() checks the placeholder shapes validate()

The list for constraint (1):

print("VPU_OPS =", sorted(short(t) for t in VPU_OPS))
VPU_OPS = ['add', 'cat', 'conv2d', 'flatten', 'linear', 'max_pool2d', 'relu']

The quantizer annotates only these ops; the partitioner in M10-3 reads the same list.

B. Scale Is a Power of 2: Pow2MinMaxObserver

Pow2MinMaxObserver overrides only calculate_qparams(): it computes the scale as MinMaxObserver does, then rounds it up to \(2^k\) (round_up_pow2). Feed the same tensor to both observers (both int8 [-128, 127], per_tensor_symmetric), apply torch.fake_quantize_per_tensor_affine with each scale, and compare SQNR against the original tensor:

model = trained_tiny_net()
imgs = make_tensors(n=64, seed=123)[0]
with torch.no_grad():
    tensors = {"model input": imgs, "stem (conv+bn+relu) output": model.stem(imgs)}
rows, ratio, loss = [], {}, {}
for name, t in tensors.items():
    result = {}
    for cls in (MinMaxObserver, Pow2MinMaxObserver):
        obs = cls(dtype=torch.int8, quant_min=-128, quant_max=127, qscheme=torch.per_tensor_symmetric)
        obs(t)
        scale = float(obs.calculate_qparams()[0])
        snr = sqnr(t, torch.fake_quantize_per_tensor_affine(t, scale, 0, -128, 127))
        result[cls.__name__] = (scale, snr)
        rows.append([name, cls.__name__, f"{scale:.6f}", f"{torch.log2(torch.tensor(scale)):.3f}", f"{127 * scale:.4f}", f"{snr:.1f}"])
    (s0, q0), (s1, q1) = result.values()
    ratio[name], loss[name] = s1 / s0, q0 - q1
table(rows, ["tensor", "observer", "scale", "log2(scale)", "127 × scale", "SQNR (dB)"])
tensor observer scale log2(scale) 127 × scale SQNR (dB)
model input MinMaxObserver 0.027411 -5.189 3.4812 40.2
model input Pow2MinMaxObserver 0.031250 -5.000 3.9688 39.1
stem (conv+bn+relu) output MinMaxObserver 0.037425 -4.740 4.7530 38.4
stem (conv+bn+relu) output Pow2MinMaxObserver 0.062500 -4.000 7.9375 34.0
print("pow2 scale / MinMax scale:", {k: round(v, 2) for k, v in ratio.items()})
print("SQNR lost (dB):", {k: round(v, 1) for k, v in loss.items()})
assert all(1 <= r < 2 for r in ratio.values()) and all(0 <= d <= 6.1 for d in loss.values())
pow2 scale / MinMax scale: {'model input': 1.14, 'stem (conv+bn+relu) output': 1.67}
SQNR lost (dB): {'model input': 1.1, 'stem (conv+bn+relu) output': 4.5}
  • It rounds up, so the pow2 scale is 1x to 2x the original: 127 × scale (the largest representable value) does not shrink, so nothing extra is clipped.
  • The cost is that every step gets coarser and SQNR drops. The scale becomes at most 2x, the resolution is at most 2x worse, and SQNR loses at most about 6 dB.
  • What you get in return: the requant factor \(s_{in} s_w / s_{out}\) is also \(2^k\), so the hardware can do it with a shift (M10-2 C).

convert_pt2e reads the scale through calculate_qparams() (M02-3), so overriding it decides the numbers on the Q/DQ graph (confirmed in section E).

C. validate(): Dynamic Batch Is Rejected Inside prepare_pt2e

By default export_module gives dynamic_shapes (dimension 0 is dynamic). Run prepare_pt2e(…, MyVPUQuantizer()) directly on that graph:

error = None
try:
    prepare_pt2e(export_module(model, example_inputs(BATCH)), MyVPUQuantizer())
except ValueError as e:
    error = str(e)
print("ValueError:", error)
assert error is not None
ValueError: Input x has shape (s77, 3, 16, 16) with a dynamic dimension; this VPU takes static shapes only, so do not pass dynamic_shapes when exporting

Because dynamic_shapes was given at export, dimension 0 of the placeholder’s shape is a SymInt (a name like s77). prepare_pt2e calls validate() after annotate() (M03-3), and validate() raises as soon as it sees a SymInt, before any observer is inserted.

Static shape cannot be written as an annotation on any tensor, so it goes in validate(). Sections D and E switch to export_module(model, x, dynamic_batch=False), that is, torch.export.export without dynamic_shapes.

D. TinyConvNet on the VPU: Which Tensors Are Forced to Use the Same Scale

x = example_inputs(BATCH)
prepared = prepare_pt2e(export_module(model, x, dynamic_batch=False), MyVPUQuantizer())
with torch.no_grad():
    for b in calibration_batches(n=64, batch_size=BATCH):  # static shape: the calibration batch must equal the export batch
        prepared(*b)

SharedQuantizationSpec makes several edges share the same observer instance (M03-4). Group the activation observers by instance; tensors in the same group necessarily have the same scale:

groups = {}  # observer instance -> the tensors it observes
for n, mod in observer_nodes(prepared):
    src = n.args[0]
    if src.op != "get_attr":  # skip the weight/bias observers
        groups.setdefault(id(mod), (mod, []))[1].append(n)
shared_mod, shared = max(groups.values(), key=lambda g: len(g[1]))
print("Largest group:", [n.args[0].name for n in shared])
assert len(shared) == 5
Largest group: ['relu', 'max_pool2d', 'conv2d_2', 'relu_2', 'max_pool2d_1']

These 5 tensors are in the residual block. Draw that part, with the ones tied to the same observer drawn with a thick border (the number printed under each observer is the calibrated scale):

draw(prepared, focus=["max_pool2d", "add", "relu_2", "conv2d_2", "max_pool2d_1"], hops=1,
     highlight=[n.name for n in shared], title="Residual block: the thick-bordered observers are the same instance")

Constraint (6) chains all the way through:

  • add requires its two inputs (the output of max_pool2d and the output of conv2d_2) and its output to share one scale; the relu_2 fused after add is its output.
  • max_pool2d requires input and output to share a scale, so its input (the stem’s relu) is pulled in too; max_pool2d_1 likewise.

The price is that tensors with a smaller range also have to use the scale decided by the largest range in the chain. Also, having the same scale does not necessarily mean being tied together:

s_shared = float(shared_mod.calculate_qparams()[0])
by_chance = [g[1][0].args[0].name for g in groups.values()
             if g[0] is not shared_mod and float(g[0].calculate_qparams()[0]) == s_shared]
print(f"Scale of the tied group: 2^{torch.log2(torch.tensor(s_shared)):.0f}; other tensors whose scale happens to be the same: {by_chance}")
Scale of the tied group: 2^-4; other tensors whose scale happens to be the same: ['relu_1', 'relu_3']

These tensors have the same scale as that chain, but they have their own observers: powers of 2 are coarse, and they just happen to land in the same bucket. With different calibration data they might differ; the tied chain is always identical.

There is no batch_norm on the graph: prepare_pt2e first folds BN into conv (M04-1). adaptive_avg_pool2d is not in VPU_OPS, gets no annotation, and stays in fp32 (M10-3 shows how it cuts the graph apart).

E. The Q/DQ Graph After convert_pt2e

converted = convert_pt2e(prepared)
q = [n for n in qdq_nodes(converted) if n.target == torch.ops.quantized_decomposed.quantize_per_tensor.default]
scales = torch.tensor([float(n.args[1]) for n in q])
pow2 = bool(torch.all(torch.log2(scales) == torch.round(torch.log2(scales))))
zp0 = all(n.args[2] == 0 for n in q)
print(f"{len(q)} quantize_per_tensor: all scales are powers of 2: {pow2}; all zero points are 0: {zp0}")
assert pow2 and zp0
11 quantize_per_tensor: all scales are powers of 2: True; all zero points are 0: True

Finally, compute one SQNR against the fp32 output, only to confirm the model still works (this does not break down how much accuracy each constraint costs). The baseline is the torchao test-version XNNPACKQuantizer with the same calibration data:

xnn = prepare_pt2e(export_module(model, x, dynamic_batch=False), get_xnnpack_quantizer(is_per_channel=True)[0])
xt = make_tensors(n=BATCH, seed=7)[0]
with torch.no_grad():
    for b in calibration_batches(n=64, batch_size=BATCH):
        xnn(*b)
    xnn = convert_pt2e(xnn)
    y = model(xt)
    s_vpu, s_xnn = sqnr(y, converted(xt)), sqnr(y, xnn(xt))
print(f"SQNR vs the fp32 output: MyVPUQuantizer {s_vpu:.1f} dB; XNNPACKQuantizer (torchao test version) {s_xnn:.1f} dB")
assert s_vpu > 15
SQNR vs the fp32 output: MyVPUQuantizer 24.6 dB; XNNPACKQuantizer (torchao test version) 28.6 dB

Summary

  • A backend Quantizer = hardware constraints translated into annotations: qscheme handles symmetric, a custom observer handles the form of the scale, Shared/Derived handle the relations between scales, “don’t annotate” handles the op list, and validate() handles conditions annotations cannot express (static shape).
  • The same-scale requirement chains along add and pool; the cost is borne by the tensors with the smaller ranges.
  • All of this is only the Quantizer’s own promise. The backend compiler receives nothing but the Q/DQ graph, so it has to check (M10-2) and partition (M10-3) on its own.