import copy
import torch
from common.data import calibration_batches
from common.graph_utils import observer_nodes, qdq_nodes
from common.models import example_inputs, get_model
from common.pt2e import (
ANNOTATION_KEY,
Quantizer,
annotate_input_qspec_map,
annotate_output_qspec,
convert_pt2e,
export_module,
get_xnnpack_quantizer,
prepare_pt2e,
quiet,
)
from common.qspecs import act_spec, weight_spec
from common.viz import annotation_notes, draw, table
quiet()
_ = torch.manual_seed(0)
CONV2D = torch.ops.aten.conv2d.defaultM03-3 | Writing a Rule-Based Quantizer from Scratch: Quantize Only conv2d
The LabelQuantizer of M03-1 works by name: we tell it “annotate the output of relu”. On a different model the node names differ, so naming doesn’t generalize. A real Quantizer finds nodes by rules:
\[ \text{Quantizer} = \text{rules for finding nodes} + \text{which spec to use at each position} \]
This notebook writes a ConvOnlyQuantizer:
- Rule: nodes whose
node.targetisaten.conv2d. - Specs: int8 per-tensor affine for the activation, int8 per-channel symmetric for the weight (
act_spec()andweight_spec()incommon/qspecs.py).
Sections:
- A. Writing it: the three actions of
annotate. - B. Comparing with XNNPACKQuantizer: the observer between conv and relu.
- C.
validate: an unsupported graph errors out at prepare time. - D. What this Quantizer still lacks.
A. Writing It: The Three Actions of annotate
For each node, annotate does three things:
- Find the node: match the op by
node.target, ignoring names. - Attach annotations: torchao provides two public helper functions,
annotate_input_qspec_map(node, upstream, spec)to mark an input edge andannotate_output_qspec(node, spec)to mark the output. They create theQuantizationAnnotationwhen needed, so you don’t construct one yourself. - Mark as handled: set
_annotated = True. Also check it at the start and skip nodes that were already annotated. When several Quantizers are composed, this is the convention for “don’t grab someone else’s node” (M03-6).
class ConvOnlyQuantizer(Quantizer):
def annotate(self, model):
for node in model.graph.nodes:
if node.op != "call_function" or node.target != CONV2D: # (1) find the node
continue
ann = node.meta.get(ANNOTATION_KEY)
if ann is not None and ann._annotated: # (3) leave already-annotated nodes alone
continue
activation, weight = node.args[0], node.args[1]
annotate_input_qspec_map(node, activation, act_spec()) # (2) attach annotations
annotate_input_qspec_map(node, weight, weight_spec())
annotate_output_qspec(node, act_spec())
node.meta[ANNOTATION_KEY]._annotated = True # (3)
return model
def validate(self, model):
"""The weight must be a model constant (get_attr); otherwise it can't be pre-quantized."""
for node in model.graph.nodes:
if node.op == "call_function" and node.target == CONV2D and node.args[1].op != "get_attr":
raise ValueError(f"The weight of {node.name} comes from the {node.args[1].op} node '{node.args[1].name}'; "
"ConvOnlyQuantizer only supports constant weights")The bias (node.args[2]) is not put into input_qspec_map, so it is not quantized. Call annotate directly and look at the annotations:
gm = export_module(get_model("conv_relu").eval(), example_inputs(4))
q = ConvOnlyQuantizer()
q.annotate(gm)
draw(gm, notes=annotation_notes(gm, unlabeled=True), title="Annotations ConvOnlyQuantizer attaches on conv → relu")Compared with the XNNPACKQuantizer of M03-2 section A, there are two differences: the bias edge isn’t in the annotations at all (XNNPACK lists it with spec None; the result is the same, not quantized); and conv2d itself has an output annotation, while relu has none.
Call annotate once more: the annotation objects aren’t replaced, because the _annotated check at the start blocks it:
before = {n.name: id(n.meta[ANNOTATION_KEY]) for n in gm.graph.nodes if ANNOTATION_KEY in n.meta}
q.annotate(gm)
after = {n.name: id(n.meta[ANNOTATION_KEY]) for n in gm.graph.nodes if ANNOTATION_KEY in n.meta}
print("After calling annotate again, are the annotation objects the same?", before == after)
assert before == afterAfter calling annotate again, are the annotation objects the same? True
B. Comparing with XNNPACKQuantizer: The Observer Between conv and relu
Hand the same conv_bn_relu model to each of the two Quantizers for prepare_pt2e (BN is folded into conv first), and draw only the area around conv and relu:
gm = export_module(get_model("conv_bn_relu").eval(), example_inputs(8))
mine = prepare_pt2e(copy.deepcopy(gm), ConvOnlyQuantizer())
draw(mine, focus=["conv2d", "relu"], title="ConvOnlyQuantizer: after prepare")xnnpack, _ = get_xnnpack_quantizer(is_per_channel=True)
theirs = prepare_pt2e(copy.deepcopy(gm), xnnpack)
draw(theirs, focus=["conv2d", "relu"], title="XNNPACKQuantizer: after prepare")def observer_before_relu(gm):
"""Whether the input of relu is an observer: if so, there is a quantization point between conv and relu."""
relu = next(n for n in gm.graph.nodes if n.target == torch.ops.aten.relu.default)
return relu.args[0].op == "call_module"
print("ConvOnlyQuantizer: observer between conv and relu?", observer_before_relu(mine))
print("XNNPACKQuantizer: observer between conv and relu?", observer_before_relu(theirs))
assert observer_before_relu(mine) and not observer_before_relu(theirs)ConvOnlyQuantizer: observer between conv and relu? True
XNNPACKQuantizer: observer between conv and relu? False
ConvOnlyQuantizer only recognizes conv and doesn’t know a relu follows, so it follows the rule and annotates the conv’s output, leaving one extra quantization point between conv and relu. XNNPACKQuantizer recognizes the conv + relu pattern and puts the output annotation on relu, so the backend can run the two ops as one kernel. Both graphs run; the difference is whether the backend can fuse.
Switch to the larger tiny model (conv, maxpool, add, pool, linear, …) and see how much each Quantizer quantizes:
def count(model_name, quantizer):
prepared = prepare_pt2e(export_module(get_model(model_name).eval(), example_inputs(8)), quantizer)
n_obs = len(observer_nodes(prepared))
with torch.no_grad():
for batch in calibration_batches(n=64, batch_size=16):
prepared(*batch)
return n_obs, len(qdq_nodes(convert_pt2e(prepared)))
rows = []
for name, quantizer in [("ConvOnlyQuantizer", ConvOnlyQuantizer()),
("XNNPACKQuantizer", get_xnnpack_quantizer(is_per_channel=True)[0])]:
rows.append([name, *count("tiny", quantizer)])
table(rows, ["Quantizer", "# observers", "# q/dq"])| Quantizer | # observers | # q/dq |
|---|---|---|
| ConvOnlyQuantizer | 12 | 20 |
| XNNPACKQuantizer | 16 | 28 |
assert rows[0][1] < rows[1][1] and rows[0][2] < rows[1][2]ConvOnlyQuantizer annotates only convs; maxpool, add, pool and linear have no annotation and stay in fp32, so there are fewer observers and q/dq. XNNPACKQuantizer annotates more kinds of ops.
C. validate: An Unsupported Graph Errors Out at prepare Time
annotate only attaches annotations; “can I handle this graph” is left to validate. prepare_pt2e calls it after annotate and before inserting observers (M03-1), so the user finds out before calibration.
The model below takes the weight as an input instead of as a model parameter. A weight that isn’t a constant can’t be pre-quantized to int8:
class DynamicWeightConv(torch.nn.Module):
def forward(self, x, w):
return torch.nn.functional.conv2d(x, w, padding=1)
bad = export_module(DynamicWeightConv(), (torch.randn(1, 3, 8, 8), torch.randn(4, 3, 3, 3)), dynamic_batch=False)
try:
prepare_pt2e(copy.deepcopy(bad), ConvOnlyQuantizer())
raised = None
except ValueError as e:
raised = e
print("ValueError:", raised)
assert raised is not NoneValueError: The weight of conv2d comes from the placeholder node 'w'; ConvOnlyQuantizer only supports constant weights
When a backend has hardware limits, validate is where they are checked (M10-1 uses it to reject dynamic batch).
D. What This Quantizer Still Lacks
| What’s missing | Consequence | Where it’s handled |
|---|---|---|
| pattern | Doesn’t know a conv followed by relu can be fused, so there are more quantization points | M03-6 annotation order, M04 fusion |
| Shared qparams | The inputs and output of add and cat each have a different scale, so a rescale is needed before integer arithmetic | M03-4 |
| bias quantization | Backends often require an int32 bias, with the scale derived from activation × weight | M03-5 |
| Config interface | The user can’t specify “which part uses which spec, which part isn’t quantized” | M03-6, M03-7 |
Summary
- A rule-based Quantizer finds ops by
node.target, so it keeps working when the model changes. - The three actions of
annotate: find nodes, attach annotations (annotate_input_qspec_map,annotate_output_qspec), mark_annotated. - A rule that recognizes only a single op leaves quantization points in the middle of patterns;
validatemakes unsupported graphs error out at prepare time.