import copy
from typing import Any, cast
import torch
from common.graph_utils import observer_nodes
from common.models import example_inputs, get_model
from common.pt2e import (
DerivedQuantizationSpec,
FixedQParamsQuantizationSpec,
Quantizer,
annotate_input_qspec_map,
annotate_output_qspec,
convert_pt2e,
export_module,
prepare_pt2e,
quiet,
)
from common.qspecs import act_spec, weight_spec
from common.viz import draw_stages, table
quiet()
_ = torch.manual_seed(0)
CONV2D = torch.ops.aten.conv2d.default
SIGMOID = torch.ops.aten.sigmoid.defaultM03-5 | qparams Without Calibration: Derived and FixedQParams
So far, the scale and zero point at every position have been computed by observers looking at data. But for some tensors the qparams shouldn’t be decided by calibration:
| spec | Where qparams come from | Example |
|---|---|---|
DerivedQuantizationSpec |
Computed from observers at other positions | conv bias: scale = activation scale × weight scale |
FixedQParamsQuantizationSpec |
Hard-coded in the spec, independent of data | Output of sigmoid: range fixed in (0, 1); a model output that must match a downstream fixed-point format (e.g. Q3.12) |
- A. Derived: the bias is quantized to int32, with the scale derived from the activation and the weight.
- B. FixedQParams: the sigmoid output uses a fixed scale unaffected by calibration data; and how to use it to align the format when the model output goes straight to a DSP doing integer arithmetic.
A. Derived: The Bias Scale Is Derived from the Activation and the Weight
Substitute \(x = s_x(q_x - z_x)\) and \(w = s_w(q_w - z_w)\) into an integer conv, where \(z_x\), \(z_w\) are the respective zero points. Expanding the multiply-accumulate of one output channel, the common factor is \(s_x s_w\) (with a per-channel weight, each output channel has its own \(s_w\)):
\[ \sum x w = s_x s_w \Big( \sum q_x q_w - z_w \sum q_x - z_x \sum q_w + N z_x z_w \Big) \]
\(N\) is the number of terms summed in this channel. The bias is quantized to int32 with the same scale and zero point 0, so that \(q_b\) can be added straight into the parentheses:
\[ y \approx s_x s_w \Big( \sum q_x q_w - z_w \sum q_x - z_x \sum q_w + N z_x z_w + q_b \Big), \qquad q_b = \operatorname{round}\!\Big(\frac{b}{s_x s_w}\Big) \tag{1} \]
\(z_x\) and \(z_w\) appear only in the integer terms inside the parentheses; no bias zero point comes out. The constant terms \(N z_x z_w\) and \(z_x \sum q_w\) can be folded separately into \(q_b\); \(z_w \sum q_x\) depends on the current input, so the weight is usually symmetric (\(z_w = 0\)). The weight_spec() below is exactly this setting. The \(q_b\) in Eq. (1) only quantizes the bias itself, without folding in those constant terms.
This is common practice in integer inference, and the actual requirement depends on the backend (the VPU example in M10 requires an int32 bias). Since the bias scale is decided by others, it can’t be calibrated on its own, only derived: the derived_from of DerivedQuantizationSpec lists the positions it depends on, and derive_qparams_fn receives the observers at those positions (in the same order as derived_from) and returns (scale, zero point).
def derive_bias_qparams(obs_or_fqs):
act_scale, _ = obs_or_fqs[0].calculate_qparams() # shape [1]
weight_scale, _ = obs_or_fqs[1].calculate_qparams() # shape [out_channels]
bias_scale = act_scale * weight_scale
return bias_scale, torch.zeros_like(bias_scale, dtype=torch.int32)
class ConvBiasQuantizer(Quantizer):
def annotate(self, model):
for node in model.graph.nodes:
if node.op != "call_function" or node.target != CONV2D:
continue
activation, weight, bias = node.args[0], node.args[1], node.args[2]
annotate_input_qspec_map(node, activation, act_spec())
annotate_input_qspec_map(node, weight, weight_spec())
annotate_input_qspec_map(node, bias, DerivedQuantizationSpec(
derived_from=[(activation, node), (weight, node)], # depends on the observers on these two edges
derive_qparams_fn=derive_bias_qparams,
dtype=torch.int32,
quant_min=-(2**31),
quant_max=2**31 - 1,
qscheme=torch.per_channel_symmetric,
ch_axis=0,
))
annotate_output_qspec(node, act_spec())
return model
def validate(self, model):
passPrepare, calibrate, then convert. The first row is the prepare graph after calibration, the second row is after convert:
model = get_model("conv_relu").eval()
prepared = prepare_pt2e(export_module(model, example_inputs(4)), ConvBiasQuantizer())
with torch.no_grad():
for _ in range(8):
prepared(torch.randn(4, 3, 16, 16))
converted = convert_pt2e(copy.deepcopy(prepared))
conv = next(n for n in prepared.graph.nodes if n.target == CONV2D)
draw_stages([(prepared, "prepare_pt2e (after calibration)"), (converted, "convert_pt2e")], focus="conv2d", per_row=1,
notes=[{conv.args[2].name: "Derived: doesn't look at data"}, {}], title="conv bias: Derived observer → int32 constant")Look at the bias path:
- After prepare, the bias also has an observer in front of it, of class
DerivedObserverOrFakeQuantize. It records no values; only whencalculate_qparams()is called does it ask the other two observers. - After convert, the bias becomes an int32 constant (the
int32in thedequantize_per_channellabel on the right), goes through a dequantize and then into the conv. The weight likewise becomes an int8 constant.
Verify Eq. (1): the scale given by the bias observer equals the activation scale × the weight scale:
# get_submodule returns nn.Module; calculate_qparams is an observer method, expressed with Any
act_obs, weight_obs, bias_obs = (cast(Any, prepared.get_submodule(a.target)) for a in conv.args[:3])
act_scale = act_obs.calculate_qparams()[0]
weight_scale = weight_obs.calculate_qparams()[0]
bias_scale = bias_obs.calculate_qparams()[0]
print("Class of the bias observer:", type(bias_obs).__name__)
print(f"Number of scales: activation {act_scale.numel()}, weight {weight_scale.numel()}, bias {bias_scale.numel()}")
print("Max difference between bias scale and activation × weight:", float((bias_scale - act_scale * weight_scale).abs().max()))
assert torch.equal(bias_scale, act_scale * weight_scale)Class of the bias observer: DerivedObserverOrFakeQuantize
Number of scales: activation 1, weight 8, bias 8
Max difference between bias scale and activation × weight: 0.0
Now take the int32 bias out of the converted graph, multiply the scale back, and compare with the original fp32 bias:
bias_int32 = next(b for name, b in converted.named_buffers() if name.startswith("_frozen_param") and b.dtype == torch.int32)
restored = bias_int32.to(torch.float32) * bias_scale
error = (restored - model.conv.bias.detach()).abs()
print(f"Shape of the int32 bias: {tuple(bias_int32.shape)}")
print(f"Max error between the restored and original bias: {float(error.max()):.2e} (max of half a scale: {float(bias_scale.max() / 2):.2e})")
assert (error <= bias_scale / 2 + 1e-12).all()Shape of the int32 bias: (8,)
Max error between the restored and original bias: 2.23e-05 (max of half a scale: 2.40e-05)
An error of no more than half a scale is exactly the rounding error of Eq. (1). \(s_x s_w\) is very small, so the int32 bias loses almost no precision.
B. FixedQParams: No Calibration When the Range Is Known or the Format Is Prescribed
The output of sigmoid always lies in (0, 1). With uint8, scale \(= 1/256\) and zero point \(= 0\) it covers \([0, 255/256]\) without looking at any data:
class SigmoidNet(torch.nn.Module):
def forward(self, x):
return torch.sigmoid(x)
class SigmoidQuantizer(Quantizer):
"""Quantize only the output of sigmoid (the input stays fp32, so the error comes only from output quantization)."""
def __init__(self, output_spec):
self.output_spec = output_spec
def annotate(self, model):
for node in model.graph.nodes:
if node.op == "call_function" and node.target == SIGMOID:
annotate_output_qspec(node, self.output_spec)
return model
def validate(self, model):
pass
fixed = FixedQParamsQuantizationSpec(
dtype=torch.uint8, scale=1 / 256, zero_point=0, quant_min=0, quant_max=255, qscheme=torch.per_tensor_affine,
)Compare with the act_spec() (MinMax) calibrated as usual. We deliberately make the calibration data range small (randn × 0.1) and the test data range much larger (randn × 3), and look at the error of the two kinds of qparams on the test data:
gm = export_module(SigmoidNet(), (torch.randn(4, 8),))
calib = [torch.randn(4, 8) * 0.1 for _ in range(8)]
test_x = torch.randn(4, 8) * 3
rows = []
for title, spec in [("Calibrated (MinMaxObserver)", act_spec()), ("Fixed qparams", fixed)]:
p = prepare_pt2e(copy.deepcopy(gm), SigmoidQuantizer(spec))
with torch.no_grad():
for x in calib:
p(x)
obs = observer_nodes(p)[0][1]
scale, zp = obs.calculate_qparams()
with torch.no_grad():
err = float((convert_pt2e(p)(test_x) - torch.sigmoid(test_x)).abs().max())
rows.append([title, type(obs).__name__, f"{float(scale):.6f}", int(zp), f"{err:.4f}"])
table(rows, ["output qparams", "observer class", "scale", "zero point", "max error on test data"])| output qparams | observer class | scale | zero point | max error on test data |
|---|---|---|---|---|
| Calibrated (MinMaxObserver) | MinMaxObserver | 0.002261 | -128 | 0.4094 |
| Fixed qparams | FixedQParamsObserver | 0.003906 | 0 | 0.0019 |
calibrated_err, fixed_err = float(rows[0][4]), float(rows[1][4])
assert fixed_err <= 1 / 512 + 1e-6 < calibrated_err- The calibration data range is small, so the sigmoid output only falls in a narrow stretch around 0.5. After MinMax extends its range to include 0, its upper bound is only slightly above 0.5; once the test data is larger, outputs beyond the upper bound are all clipped, giving a large error.
- Fixed qparams never look at data, so the error is only rounding, at most half a scale (\(1/512\)).
This suits ops whose output range is mathematically known: sigmoid (0, 1), tanh (−1, 1), softmax (0, 1). Inference (not verified here): integer hardware often implements such ops as lookup tables, and fixed qparams keep the table contents independent of calibration data.
Another use: the output format is prescribed downstream
The qparams of sigmoid are decided by its mathematical range. Another situation is that they are decided by the downstream format: the model output goes directly to a program that does only integer arithmetic. For example, the spectrogram (or mask) output by a speech enhancement model is handed to a DSP downstream that works in fixed-point, and both sides agree in advance that the output is Q3.12:
- Q3.12 is a 16-bit signed fixed-point number: 1 sign bit, 3 integer bits, 12 fractional bits, and the integer \(q\) represents \(q / 2^{12}\).
- In affine quantization parameters: int16, scale \(= 2^{-12}\), zero point \(= 0\), integer range \([-32768, 32767]\). It can represent \([-8,\ 8 - 2^{-12}]\), with a step of \(2^{-12} \approx 0.000244\).
These qparams are an interface specification and can’t be left to calibration: the scale computed by calibration changes with the data, while the DSP side has \(2^{-12}\) hard-coded. So attach a FixedQParamsQuantizationSpec to the output of the model’s last op, filled in with the parameters above. This int16 spec works fine for both prepare and convert on torchao 0.18. Note three things:
- Values outside \([-8, 8)\) are clamped. Fixed qparams do not widen the range when the data gets larger, so first confirm the model output really falls within the range; if not, change the agreement with the DSP (e.g. Q4.11: double the range, each step twice as coarse). The observer during calibration can be used to check the range, but doesn’t decide the qparams.
- In the Q/DQ graph after convert, the last node is still a dequantize. The graph output is fp32, just with every value landing exactly on the \(2^{-12}\) grid. To hand over a real int16, the backend has to drop the final dq during lowering, or multiply by \(2^{12}\) and convert to integers before handing to the DSP; the values are exactly the same.
- In QAT, the same spec becomes a
FixedQParamsFakeQuantize, so during training the model sees the Q3.12 rounding and clamping, and can learn to keep its output within the range.
Summary
DerivedQuantizationSpec: the qparams are computed byderive_qparams_fnfrom observers at other positions; the typical use is the int32 bias, with scale \(= s_x s_w\).FixedQParamsQuantizationSpec: the qparams are hard-coded in the spec, unaffected by calibration data. Good for ops with a known range, and also when the output format is prescribed downstream (e.g. Q3.12 for a fixed-point DSP).- An observer is still inserted at both kinds of positions (
DerivedObserverOrFakeQuantize,FixedQParamsObserver); it’s just that the qparams are not decided by the data it sees.