M03 Overview | Who Decides Where Observers Go: Annotations and the Quantizer
In M02 we inserted observers by hand, copying the positions straight from XNNPACKQuantizer. A real backend can’t work that way: it has to decide, for any model, where to quantize and with what spec. That is the job of the Quantizer. This chapter opens it up and ends by writing a Quantizer of our own. The seven notebooks that follow (M03-1 to M03-7) verify each point hands-on.
Why We Need a Quantizer
When a conv is followed by a relu, should we quantize in between? Should the two inputs of an add share one scale? Should the bias be quantized to int32? These are all backend decisions: which ops the hardware can fuse and how integer addition works differ from backend to backend. PT2E concentrates these decisions in the Quantizer, so prepare_pt2e and convert_pt2e themselves don’t need to know anything about any backend.
This chapter answers four questions:
| Question | One-line answer | Verified in |
|---|---|---|
How does a Quantizer tell prepare_pt2e where to put things? |
It annotates nodes; prepare_pt2e inserts observers according to the annotations |
M03-1, M03-2 |
| How does a Quantizer find the right ops in any model? | With rules (node.target, patterns), not by naming nodes |
M03-3, M03-6 |
| How are qparams at different positions linked? | They share one observer, are derived from another position, or are hard-coded | M03-4, M03-5 |
| How does a user say which part uses which spec? | The backend’s own config API, which is rules underneath | M03-6, M03-7 |
An Example First: What XNNPACKQuantizer Does on conv → relu
The first stage is the graph after the Quantizer has annotated it, with each node’s annotation written underneath: one line “edge upstream: spec” per input edge, and one line “output: spec” for the output. The second stage is the graph after prepare_pt2e; the green boxes are the newly grown observers. Look for two things:
- Every annotation became an observer, one to one. The bias spec is “not quantized”, so it has no observer.
- The output of conv2d has no annotation, so there is no observer between conv and relu: XNNPACKQuantizer treats conv + relu as one pattern that the backend can fuse into a single kernel.
Definitions
Three roles
| Role | What it is | What it does |
|---|---|---|
Annotation QuantizationAnnotation |
Data stored in node.meta["quantization_annotation"] |
Records which input edges of this node, or its output, are to be quantized, and with what spec |
Quantizer |
A class written by the backend author | Finds nodes in the graph, annotates them, checks whether the graph can be handled |
prepare_pt2e |
A torchao function | Calls the Quantizer, then inserts observers according to the annotations; does nothing where there is no annotation |
Annotations: two kinds of position
QuantizationAnnotation describes positions with two fields:
input_qspec_map:{upstream node: spec}, attached to the consumer, meaning the edge “upstream → this node” is quantized. It only affects this one consumer.output_qspec: the output of this node is quantized. It affects all consumers of the output.
Two more fields: _annotated (already handled, other Quantizers should leave it alone; M03-3, M03-6) and allow_implicit_sharing (M03-4).
Specs: QuantizationSpec
The spec at each position is a QuantizationSpec, which answers “what should it be quantized to, and which observer collects the statistics”. Here are the activation and weight specs of XNNPACKQuantizer from the example above:
cfg = xnnpack.global_config
assert cfg is not None and cfg.input_activation is not None and cfg.weight is not None
rows = []
for field in ("dtype", "quant_min", "quant_max", "qscheme", "ch_axis", "observer_or_fake_quant_ctr"):
values = []
for spec in (cfg.input_activation, cfg.weight):
v = getattr(spec, field)
v = getattr(getattr(v, "p", None), "func", v) # observer wrapped by with_args: unwrap to get the class
values.append(getattr(v, "__name__", str(v)))
rows.append([field, *values])
table(rows, ["field", "activation", "weight"])| field | activation | weight |
|---|---|---|
| dtype | torch.int8 | torch.int8 |
| quant_min | -128 | -127 |
| quant_max | 127 | 127 |
| qscheme | torch.per_tensor_affine | torch.per_channel_symmetric |
| ch_axis | None | 0 |
| observer_or_fake_quant_ctr | HistogramObserver | PerChannelMinMaxObserver |
The activation is int8 per-tensor affine; the weight is int8 per-channel symmetric, with one scale per output channel (ch_axis=0). Which observer to use is a choice made by this XNNPACKQuantizer default config (M02-1 compared the various observers).
Besides the ordinary QuantizationSpec there are three more specs. They differ in where the qparams come from:
| spec | Where qparams come from | Example | notebook |
|---|---|---|---|
QuantizationSpec |
Its own observer, computed from data | Most activations and weights | M03-1 |
SharedQuantizationSpec(position) |
Shares one observer with another position | Inputs and output of add and cat use the same scale | M03-4 |
DerivedQuantizationSpec |
Derived from observers at other positions | int32 bias: scale = activation scale × weight scale | M03-5 |
FixedQParamsQuantizationSpec |
Hard-coded in the spec | Output of sigmoid (known range); model output handed to a fixed-point DSP (format prescribed) | M03-5 |
The two uses of FixedQParamsQuantizationSpec have one thing in common: the qparams are known in advance and need not be estimated from data. An example of the second: the spectrogram output by a speech enhancement model is handed directly to a DSP that works in fixed-point, and the two sides agree on the format Q3.12 (int16, scale \(2^{-12}\)). The scale has to be hard-coded per that agreement, and can’t be left to calibration.
The Quantizer’s three methods and the order in prepare_pt2e
| Method | Must implement? | What it does |
|---|---|---|
transform_for_annotation(model) |
No (the base class by default doesn’t change the graph) | Rewrites the graph before annotation, e.g. turns scalar constants into tensor attributes |
annotate(model) |
Yes | Finds nodes, annotates them |
validate(model) |
Yes | After annotation, checks whether the graph can be handled, and raises an error if not |
prepare_pt2e does the following in order (torchao 0.18): fold BN into conv → transform_for_annotation → annotate → validate → insert observers according to the annotations. So the Quantizer sees a graph whose BN has already been folded (M03-1 section B verifies the call order, and M03-2 uses this fact to get the annotated graph).
prepare_qat_pt2e, used for QAT, has a different order: transform_for_annotation → annotate → validate, and only then does it handle conv + BN (_fuse_conv_bn_qat); also the BN is not folded, but stays in the graph and keeps training. So under QAT the Quantizer sees a graph whose BN is not folded (what BN looks like in a QAT graph is covered in M04-2).
Two ways to find nodes
- By name: use the node name, e.g. “the output of
relu”. It breaks when the model changes, so it is only good for experiments (M03-1, M03-4 section B). - By rule: match ops with
node.target, looking at context when necessary (e.g. the only user of the conv is relu). It keeps working when the model changes, and real Quantizers all do this (M03-3).
Some annotators in the torchao test-version XNNPACKQuantizer rely on node.meta["source_fn_stack"] to find patterns. torch 2.14’s export no longer fills this field, so cat is silently left unannotated (M03-2 section C).
Verify with Code: A Ten-Line Rule-Based Quantizer
It annotates only the output of every relu. annotate_output_qspec is a public helper provided by torchao that creates the annotation when needed:
class ReluOutputQuantizer(Quantizer):
def annotate(self, model):
for node in model.graph.nodes:
if node.op == "call_function" and node.target == torch.ops.aten.relu.default:
annotate_output_qspec(node, act_spec())
node.meta[ANNOTATION_KEY]._annotated = True
return model
def validate(self, model):
pass
for name in ("conv_relu", "residual", "tiny"):
p = prepare_pt2e(export_module(get_model(name).eval(), example_inputs(4)), ReluOutputQuantizer())
n_relu = sum(n.target == torch.ops.aten.relu.default for n in p.graph.nodes)
print(f"{name:<10} {n_relu} relu → {len(observer_nodes(p))} observers")
assert len(observer_nodes(p)) == n_reluconv_relu 1 relu → 1 observers
residual 2 relu → 2 observers
tiny 4 relu → 4 observers
The same Quantizer doesn’t need to know what the model looks like; on all three models the number of observers equals the number of relus.
Summary and Chapter Map
prepare_pt2eitself doesn’t know where to quantize; it only inserts observers according to annotations. Positions and specs are all decided by the Quantizer.- Annotations have two kinds of position (input edge, output); there are four specs, differing in where the qparams come from.
- A Quantizer finds nodes by rules; fusion, sharing and partial quantization are just different ways of writing “how the annotations are attached”.
| notebook | Question it answers |
|---|---|
| M03-1 Annotations and prepare_pt2e | What an annotation looks like, how the three roles divide the work, edge annotations vs. output annotations |
| M03-2 Reading XNNPACKQuantizer’s annotations | What a real Quantizer annotates, what fusion looks like in annotations, which ops are left unannotated |
| M03-3 A rule-based Quantizer | Finding ops with node.target, _annotated, validate |
| M03-4 Shared qparams | SharedQuantizationSpec, when implicit sharing merges and when it chains |
| M03-5 Derived and FixedQParams | How the int32 bias scale is derived; using fixed qparams when the range is known or the output format is prescribed (e.g. Q3.12) |
| M03-6 Partial quantization and composition | Quantizing only part of a model, ComposableQuantizer, annotation order |
| M03-7 A backend’s config API | X86InductorQuantizer’s priority order, a setting XNNPACKQuantizer silently ignores |
What this chapter deliberately skips: how BN is folded into conv, and fusion on QAT graphs (M04); which layers should stay in fp32 (M05); writing a complete Quantizer from scratch for an imaginary piece of hardware (M10).
References
- torchao 0.18 source:
quantizer/quantizer.py(annotations and the four specs),prepare.py(inserting observers from annotations, implicit sharing),quantizer/composable_quantizer.py,quantizer/x86_inductor_quantizer.py - Official PyTorch tutorial: How to Write a Quantizer for PyTorch 2 Export Quantization
- Some basic concepts of PyTorch FX Graph / Node (in Chinese): basics such as
node.target,node.meta,users