Skip to main content

M03-3 | Writing a Rule-Based Quantizer from Scratch: Quantize Only conv2d

Open In Colab

The LabelQuantizer of M03-1 works by name: we tell it “annotate the output of relu”. On a different model the node names differ, so naming doesn’t generalize. A real Quantizer finds nodes by rules:

\[ \text{Quantizer} = \text{rules for finding nodes} + \text{which spec to use at each position} \]

This notebook writes a ConvOnlyQuantizer:

Sections:

import copy

import torch

from common.data import calibration_batches
from common.graph_utils import observer_nodes, qdq_nodes
from common.models import example_inputs, get_model
from common.pt2e import (
    ANNOTATION_KEY,
    Quantizer,
    annotate_input_qspec_map,
    annotate_output_qspec,
    convert_pt2e,
    export_module,
    get_xnnpack_quantizer,
    prepare_pt2e,
    quiet,
)
from common.qspecs import act_spec, weight_spec
from common.viz import annotation_notes, draw, table

quiet()
_ = torch.manual_seed(0)
CONV2D = torch.ops.aten.conv2d.default

A. Writing It: The Three Actions of annotate

For each node, annotate does three things:

  1. Find the node: match the op by node.target, ignoring names.
  2. Attach annotations: torchao provides two public helper functions, annotate_input_qspec_map(node, upstream, spec) to mark an input edge and annotate_output_qspec(node, spec) to mark the output. They create the QuantizationAnnotation when needed, so you don’t construct one yourself.
  3. Mark as handled: set _annotated = True. Also check it at the start and skip nodes that were already annotated. When several Quantizers are composed, this is the convention for “don’t grab someone else’s node” (M03-6).
class ConvOnlyQuantizer(Quantizer):
    def annotate(self, model):
        for node in model.graph.nodes:
            if node.op != "call_function" or node.target != CONV2D:  # (1) find the node
                continue
            ann = node.meta.get(ANNOTATION_KEY)
            if ann is not None and ann._annotated:  # (3) leave already-annotated nodes alone
                continue
            activation, weight = node.args[0], node.args[1]
            annotate_input_qspec_map(node, activation, act_spec())  # (2) attach annotations
            annotate_input_qspec_map(node, weight, weight_spec())
            annotate_output_qspec(node, act_spec())
            node.meta[ANNOTATION_KEY]._annotated = True  # (3)
        return model

    def validate(self, model):
        """The weight must be a model constant (get_attr); otherwise it can't be pre-quantized."""
        for node in model.graph.nodes:
            if node.op == "call_function" and node.target == CONV2D and node.args[1].op != "get_attr":
                raise ValueError(f"The weight of {node.name} comes from the {node.args[1].op} node '{node.args[1].name}'; "
                                 "ConvOnlyQuantizer only supports constant weights")

The bias (node.args[2]) is not put into input_qspec_map, so it is not quantized. Call annotate directly and look at the annotations:

gm = export_module(get_model("conv_relu").eval(), example_inputs(4))
q = ConvOnlyQuantizer()
q.annotate(gm)
draw(gm, notes=annotation_notes(gm, unlabeled=True), title="Annotations ConvOnlyQuantizer attaches on conv → relu")

Compared with the XNNPACKQuantizer of M03-2 section A, there are two differences: the bias edge isn’t in the annotations at all (XNNPACK lists it with spec None; the result is the same, not quantized); and conv2d itself has an output annotation, while relu has none.

Call annotate once more: the annotation objects aren’t replaced, because the _annotated check at the start blocks it:

before = {n.name: id(n.meta[ANNOTATION_KEY]) for n in gm.graph.nodes if ANNOTATION_KEY in n.meta}
q.annotate(gm)
after = {n.name: id(n.meta[ANNOTATION_KEY]) for n in gm.graph.nodes if ANNOTATION_KEY in n.meta}
print("After calling annotate again, are the annotation objects the same?", before == after)
assert before == after
After calling annotate again, are the annotation objects the same? True

B. Comparing with XNNPACKQuantizer: The Observer Between conv and relu

Hand the same conv_bn_relu model to each of the two Quantizers for prepare_pt2e (BN is folded into conv first), and draw only the area around conv and relu:

gm = export_module(get_model("conv_bn_relu").eval(), example_inputs(8))
mine = prepare_pt2e(copy.deepcopy(gm), ConvOnlyQuantizer())
draw(mine, focus=["conv2d", "relu"], title="ConvOnlyQuantizer: after prepare")

xnnpack, _ = get_xnnpack_quantizer(is_per_channel=True)
theirs = prepare_pt2e(copy.deepcopy(gm), xnnpack)
draw(theirs, focus=["conv2d", "relu"], title="XNNPACKQuantizer: after prepare")

def observer_before_relu(gm):
    """Whether the input of relu is an observer: if so, there is a quantization point between conv and relu."""
    relu = next(n for n in gm.graph.nodes if n.target == torch.ops.aten.relu.default)
    return relu.args[0].op == "call_module"


print("ConvOnlyQuantizer: observer between conv and relu?", observer_before_relu(mine))
print("XNNPACKQuantizer:  observer between conv and relu?", observer_before_relu(theirs))
assert observer_before_relu(mine) and not observer_before_relu(theirs)
ConvOnlyQuantizer: observer between conv and relu? True
XNNPACKQuantizer:  observer between conv and relu? False

ConvOnlyQuantizer only recognizes conv and doesn’t know a relu follows, so it follows the rule and annotates the conv’s output, leaving one extra quantization point between conv and relu. XNNPACKQuantizer recognizes the conv + relu pattern and puts the output annotation on relu, so the backend can run the two ops as one kernel. Both graphs run; the difference is whether the backend can fuse.

Switch to the larger tiny model (conv, maxpool, add, pool, linear, …) and see how much each Quantizer quantizes:

def count(model_name, quantizer):
    prepared = prepare_pt2e(export_module(get_model(model_name).eval(), example_inputs(8)), quantizer)
    n_obs = len(observer_nodes(prepared))
    with torch.no_grad():
        for batch in calibration_batches(n=64, batch_size=16):
            prepared(*batch)
    return n_obs, len(qdq_nodes(convert_pt2e(prepared)))


rows = []
for name, quantizer in [("ConvOnlyQuantizer", ConvOnlyQuantizer()),
                        ("XNNPACKQuantizer", get_xnnpack_quantizer(is_per_channel=True)[0])]:
    rows.append([name, *count("tiny", quantizer)])
table(rows, ["Quantizer", "# observers", "# q/dq"])
Quantizer # observers # q/dq
ConvOnlyQuantizer 12 20
XNNPACKQuantizer 16 28
assert rows[0][1] < rows[1][1] and rows[0][2] < rows[1][2]

ConvOnlyQuantizer annotates only convs; maxpool, add, pool and linear have no annotation and stay in fp32, so there are fewer observers and q/dq. XNNPACKQuantizer annotates more kinds of ops.

C. validate: An Unsupported Graph Errors Out at prepare Time

annotate only attaches annotations; “can I handle this graph” is left to validate. prepare_pt2e calls it after annotate and before inserting observers (M03-1), so the user finds out before calibration.

The model below takes the weight as an input instead of as a model parameter. A weight that isn’t a constant can’t be pre-quantized to int8:

class DynamicWeightConv(torch.nn.Module):
    def forward(self, x, w):
        return torch.nn.functional.conv2d(x, w, padding=1)


bad = export_module(DynamicWeightConv(), (torch.randn(1, 3, 8, 8), torch.randn(4, 3, 3, 3)), dynamic_batch=False)
try:
    prepare_pt2e(copy.deepcopy(bad), ConvOnlyQuantizer())
    raised = None
except ValueError as e:
    raised = e
print("ValueError:", raised)
assert raised is not None
ValueError: The weight of conv2d comes from the placeholder node 'w'; ConvOnlyQuantizer only supports constant weights

When a backend has hardware limits, validate is where they are checked (M10-1 uses it to reject dynamic batch).

D. What This Quantizer Still Lacks

What’s missing Consequence Where it’s handled
pattern Doesn’t know a conv followed by relu can be fused, so there are more quantization points M03-6 annotation order, M04 fusion
Shared qparams The inputs and output of add and cat each have a different scale, so a rescale is needed before integer arithmetic M03-4
bias quantization Backends often require an int32 bias, with the scale derived from activation × weight M03-5
Config interface The user can’t specify “which part uses which spec, which part isn’t quantized” M03-6, M03-7

Summary

  • A rule-based Quantizer finds ops by node.target, so it keeps working when the model changes.
  • The three actions of annotate: find nodes, attach annotations (annotate_input_qspec_map, annotate_output_qspec), mark _annotated.
  • A rule that recognizes only a single op leaves quantization points in the middle of patterns; validate makes unsupported graphs error out at prepare time.