Skip to main content

M03 Overview | Who Decides Where Observers Go: Annotations and the Quantizer

Open In Colab

In M02 we inserted observers by hand, copying the positions straight from XNNPACKQuantizer. A real backend can’t work that way: it has to decide, for any model, where to quantize and with what spec. That is the job of the Quantizer. This chapter opens it up and ends by writing a Quantizer of our own. The seven notebooks that follow (M03-1 to M03-7) verify each point hands-on.

Why We Need a Quantizer

When a conv is followed by a relu, should we quantize in between? Should the two inputs of an add share one scale? Should the bias be quantized to int32? These are all backend decisions: which ops the hardware can fuse and how integer addition works differ from backend to backend. PT2E concentrates these decisions in the Quantizer, so prepare_pt2e and convert_pt2e themselves don’t need to know anything about any backend.

This chapter answers four questions:

Question One-line answer Verified in
How does a Quantizer tell prepare_pt2e where to put things? It annotates nodes; prepare_pt2e inserts observers according to the annotations M03-1, M03-2
How does a Quantizer find the right ops in any model? With rules (node.target, patterns), not by naming nodes M03-3, M03-6
How are qparams at different positions linked? They share one observer, are derived from another position, or are hard-coded M03-4, M03-5
How does a user say which part uses which spec? The backend’s own config API, which is rules underneath M03-6, M03-7

An Example First: What XNNPACKQuantizer Does on conv → relu

The first stage is the graph after the Quantizer has annotated it, with each node’s annotation written underneath: one line “edge upstream: spec” per input edge, and one line “output: spec” for the output. The second stage is the graph after prepare_pt2e; the green boxes are the newly grown observers. Look for two things:

  • Every annotation became an observer, one to one. The bias spec is “not quantized”, so it has no observer.
  • The output of conv2d has no annotation, so there is no observer between conv and relu: XNNPACKQuantizer treats conv + relu as one pattern that the backend can fuse into a single kernel.

Definitions

Three roles

Role What it is What it does
Annotation QuantizationAnnotation Data stored in node.meta["quantization_annotation"] Records which input edges of this node, or its output, are to be quantized, and with what spec
Quantizer A class written by the backend author Finds nodes in the graph, annotates them, checks whether the graph can be handled
prepare_pt2e A torchao function Calls the Quantizer, then inserts observers according to the annotations; does nothing where there is no annotation

Annotations: two kinds of position

QuantizationAnnotation describes positions with two fields:

  • input_qspec_map: {upstream node: spec}, attached to the consumer, meaning the edge “upstream → this node” is quantized. It only affects this one consumer.
  • output_qspec: the output of this node is quantized. It affects all consumers of the output.

Two more fields: _annotated (already handled, other Quantizers should leave it alone; M03-3, M03-6) and allow_implicit_sharing (M03-4).

Specs: QuantizationSpec

The spec at each position is a QuantizationSpec, which answers “what should it be quantized to, and which observer collects the statistics”. Here are the activation and weight specs of XNNPACKQuantizer from the example above:

cfg = xnnpack.global_config
assert cfg is not None and cfg.input_activation is not None and cfg.weight is not None
rows = []
for field in ("dtype", "quant_min", "quant_max", "qscheme", "ch_axis", "observer_or_fake_quant_ctr"):
    values = []
    for spec in (cfg.input_activation, cfg.weight):
        v = getattr(spec, field)
        v = getattr(getattr(v, "p", None), "func", v)  # observer wrapped by with_args: unwrap to get the class
        values.append(getattr(v, "__name__", str(v)))
    rows.append([field, *values])
table(rows, ["field", "activation", "weight"])
field activation weight
dtype torch.int8 torch.int8
quant_min -128 -127
quant_max 127 127
qscheme torch.per_tensor_affine torch.per_channel_symmetric
ch_axis None 0
observer_or_fake_quant_ctr HistogramObserver PerChannelMinMaxObserver

The activation is int8 per-tensor affine; the weight is int8 per-channel symmetric, with one scale per output channel (ch_axis=0). Which observer to use is a choice made by this XNNPACKQuantizer default config (M02-1 compared the various observers).

Besides the ordinary QuantizationSpec there are three more specs. They differ in where the qparams come from:

spec Where qparams come from Example notebook
QuantizationSpec Its own observer, computed from data Most activations and weights M03-1
SharedQuantizationSpec(position) Shares one observer with another position Inputs and output of add and cat use the same scale M03-4
DerivedQuantizationSpec Derived from observers at other positions int32 bias: scale = activation scale × weight scale M03-5
FixedQParamsQuantizationSpec Hard-coded in the spec Output of sigmoid (known range); model output handed to a fixed-point DSP (format prescribed) M03-5

The two uses of FixedQParamsQuantizationSpec have one thing in common: the qparams are known in advance and need not be estimated from data. An example of the second: the spectrogram output by a speech enhancement model is handed directly to a DSP that works in fixed-point, and the two sides agree on the format Q3.12 (int16, scale \(2^{-12}\)). The scale has to be hard-coded per that agreement, and can’t be left to calibration.

The Quantizer’s three methods and the order in prepare_pt2e

Method Must implement? What it does
transform_for_annotation(model) No (the base class by default doesn’t change the graph) Rewrites the graph before annotation, e.g. turns scalar constants into tensor attributes
annotate(model) Yes Finds nodes, annotates them
validate(model) Yes After annotation, checks whether the graph can be handled, and raises an error if not

prepare_pt2e does the following in order (torchao 0.18): fold BN into conv → transform_for_annotation → annotate → validate → insert observers according to the annotations. So the Quantizer sees a graph whose BN has already been folded (M03-1 section B verifies the call order, and M03-2 uses this fact to get the annotated graph).

prepare_qat_pt2e, used for QAT, has a different order: transform_for_annotation → annotate → validate, and only then does it handle conv + BN (_fuse_conv_bn_qat); also the BN is not folded, but stays in the graph and keeps training. So under QAT the Quantizer sees a graph whose BN is not folded (what BN looks like in a QAT graph is covered in M04-2).

Two ways to find nodes

  • By name: use the node name, e.g. “the output of relu”. It breaks when the model changes, so it is only good for experiments (M03-1, M03-4 section B).
  • By rule: match ops with node.target, looking at context when necessary (e.g. the only user of the conv is relu). It keeps working when the model changes, and real Quantizers all do this (M03-3).

Some annotators in the torchao test-version XNNPACKQuantizer rely on node.meta["source_fn_stack"] to find patterns. torch 2.14’s export no longer fills this field, so cat is silently left unannotated (M03-2 section C).

Verify with Code: A Ten-Line Rule-Based Quantizer

It annotates only the output of every relu. annotate_output_qspec is a public helper provided by torchao that creates the annotation when needed:

class ReluOutputQuantizer(Quantizer):
    def annotate(self, model):
        for node in model.graph.nodes:
            if node.op == "call_function" and node.target == torch.ops.aten.relu.default:
                annotate_output_qspec(node, act_spec())
                node.meta[ANNOTATION_KEY]._annotated = True
        return model

    def validate(self, model):
        pass


for name in ("conv_relu", "residual", "tiny"):
    p = prepare_pt2e(export_module(get_model(name).eval(), example_inputs(4)), ReluOutputQuantizer())
    n_relu = sum(n.target == torch.ops.aten.relu.default for n in p.graph.nodes)
    print(f"{name:<10} {n_relu} relu → {len(observer_nodes(p))} observers")
    assert len(observer_nodes(p)) == n_relu
conv_relu  1 relu → 1 observers
residual   2 relu → 2 observers
tiny       4 relu → 4 observers

The same Quantizer doesn’t need to know what the model looks like; on all three models the number of observers equals the number of relus.

Summary and Chapter Map

  • prepare_pt2e itself doesn’t know where to quantize; it only inserts observers according to annotations. Positions and specs are all decided by the Quantizer.
  • Annotations have two kinds of position (input edge, output); there are four specs, differing in where the qparams come from.
  • A Quantizer finds nodes by rules; fusion, sharing and partial quantization are just different ways of writing “how the annotations are attached”.
notebook Question it answers
M03-1 Annotations and prepare_pt2e What an annotation looks like, how the three roles divide the work, edge annotations vs. output annotations
M03-2 Reading XNNPACKQuantizer’s annotations What a real Quantizer annotates, what fusion looks like in annotations, which ops are left unannotated
M03-3 A rule-based Quantizer Finding ops with node.target, _annotated, validate
M03-4 Shared qparams SharedQuantizationSpec, when implicit sharing merges and when it chains
M03-5 Derived and FixedQParams How the int32 bias scale is derived; using fixed qparams when the range is known or the output format is prescribed (e.g. Q3.12)
M03-6 Partial quantization and composition Quantizing only part of a model, ComposableQuantizer, annotation order
M03-7 A backend’s config API X86InductorQuantizer’s priority order, a setting XNNPACKQuantizer silently ignores

What this chapter deliberately skips: how BN is folded into conv, and fusion on QAT graphs (M04); which layers should stay in fp32 (M05); writing a complete Quantizer from scratch for an imaginary piece of hardware (M10).

References