import copy
import dataclasses
from collections import OrderedDict
import torch
from common.data import calibration_batches
from common.graph_utils import observer_nodes, qdq_nodes
from common.models import ConvBnReLU, example_inputs
from common.pt2e import (
X86InductorQuantizer,
convert_pt2e,
export_module,
get_default_x86_inductor_quantization_config,
get_module_name_filter,
get_xnnpack_api,
prepare_pt2e,
quiet,
)
from common.viz import table
quiet()
_ = torch.manual_seed(0)
CONV2D = torch.ops.aten.conv2d.defaultM03-7 | A Backend’s Config API: global, Op Type, Module Name
In M03-6 we wrote the rules ourselves (node.target + module name filter) to decide which nodes to quantize. Real backends wrap such rules in a config API for users to call. Take X86InductorQuantizer (torchao’s Quantizer for x86 CPU + Inductor) as an example:
| Call | Applies to |
|---|---|
set_global(cfg) |
Nodes not hit by any other rule |
set_module_type_qconfig(nn.Conv2d, cfg) |
Ops produced by a certain module class |
set_module_name_qconfig("stem", cfg) |
Ops produced by the submodule of a certain name |
A cfg of None means this part is not quantized. These set_* methods are not part of the Quantizer base class API; each backend defines its own, and names and behavior vary by backend:
- A.
X86InductorQuantizer: the priority order of the three settings, and excluding a section withNone. - B. The same idea on the torchao test-version XNNPACKQuantizer:
set_module_namesilently has no effect on conv.
The model is three conv + bn + relu segments in series, with submodule names stem, block, head_conv:
chain = torch.nn.Sequential(OrderedDict(stem=ConvBnReLU(3, 8), block=ConvBnReLU(8, 8), head_conv=ConvBnReLU(8, 8)))
gm = export_module(chain.eval(), example_inputs(4))
convs = [n.name for n in gm.graph.nodes if n.target == CONV2D]
print("convs in the graph:", convs)convs in the graph: ['conv2d', 'conv2d_1', 'conv2d_2']
A. X86InductorQuantizer: Whose Priority Is Higher
To know which setting each conv actually got, make a fingerprint for each setting: copy the default config and change only the activation’s quant_max to a different value (global 255, op type 200, module name 100). After prepare, read the quant_max of the observer on the conv input to see which one took effect. These configs are only for telling them apart, so precision isn’t compared here.
g = get_default_x86_inductor_quantization_config()
def with_qmax(qmax):
"""Copy the default config, changing only the activation's quant_max."""
act = g.input_activation
assert act is not None
return dataclasses.replace(g, input_activation=dataclasses.replace(act, quant_max=qmax))
by_type, by_name = with_qmax(200), with_qmax(100)
fingerprint = {255: "global", 200: "op type", 100: "module name"}
def make(use_global=True, type_cfg=None, names=None):
q = X86InductorQuantizer()
if use_global:
q.set_global(g)
if type_cfg:
q.set_module_type_qconfig(torch.nn.Conv2d, type_cfg)
for name, cfg in (names or {}).items():
q.set_module_name_qconfig(name, cfg)
return q
def applied(prepared):
"""The setting each conv got: look at the quant_max of its input observer; no observer means fp32."""
result = []
for conv in (n for n in prepared.graph.nodes if n.target == CONV2D):
src = conv.args[0]
result.append(fingerprint[getattr(prepared.get_submodule(src.target), "quant_max")]
if src.op == "call_module" else "fp32")
return result
settings = [
("set_global only", make()),
("global + op type (nn.Conv2d)", make(type_cfg=by_type)),
("global + op type + module name (stem)", make(type_cfg=by_type, names={"stem": by_name})),
("global + module name (block) = None", make(names={"block": None})),
("no global, module name (stem) only", make(use_global=False, names={"stem": by_name})),
]
rows = []
for title, quantizer in settings:
prepared = prepare_pt2e(copy.deepcopy(gm), quantizer)
used = applied(prepared)
with torch.no_grad():
for batch in calibration_batches(n=16, batch_size=16):
prepared(*batch)
rows.append([title, *used, len(qdq_nodes(convert_pt2e(prepared)))])
table(rows, ["setting", "stem", "block", "head_conv", "# q/dq"])| setting | stem | block | head_conv | # q/dq |
|---|---|---|---|---|
| set_global only | global | global | global | 9 |
| global + op type (nn.Conv2d) | op type | op type | op type | 9 |
| global + op type + module name (stem) | module name | op type | op type | 9 |
| global + module name (block) = None | global | fp32 | global | 6 |
| no global, module name (stem) only | module name | fp32 | fp32 | 3 |
assert [r[1:4] for r in rows] == [
["global"] * 3,
["op type"] * 3,
["module name", "op type", "op type"],
["global", "fp32", "global"],
["module name", "fp32", "fp32"],
]- Priority: module name > op type > global (rows 2 and 3). A node is annotated only once, by the first rule that hits it, and later stages skip it when they see it is already annotated (see the description of
X86InductorQuantizer.annotate); it is the same mechanism as_annotatedin M03-6. - Setting
Noneexplicitly leaves this section unquantized (row 4); this is the most direct way to write partial quantization. - Without a global, nodes not hit by any rule are not quantized (row 5).
Method names differ by backend; for example X86/Arm Inductor use set_module_name_qconfig, while XNNPACK uses set_module_name. The rules we wrote ourselves in M03-6 (node.target + module name filter) are what’s behind these APIs.
B. XNNPACKQuantizer’s set_module_name Has No Effect on conv
Do the same thing with the torchao test-version XNNPACKQuantizer: set_module_name("stem", cfg), expecting stem to be quantized.
xnnpack_cls, config_fn, _ = get_xnnpack_api()
prepared = prepare_pt2e(copy.deepcopy(gm), xnnpack_cls().set_module_name("stem", config_fn(is_per_channel=True)))
print("Number of observers after prepare:", len(observer_nodes(prepared)))
assert len(observer_nodes(prepared)) == 0Number of observers after prepare: 0
Not a single observer: stem silently stays in fp32, with no error. The cause is in the filter function: it requires every node in the pattern to belong to stem. The conv itself has an nn_module_stack, but the conv’s weight is a get_attr node, which has no such field:
in_stem = get_module_name_filter("stem")
conv = next(n for n in gm.graph.nodes if n.target == CONV2D)
print(f"Does conv belong to stem? {in_stem(conv)}")
print(f"Does the conv's weight ({conv.args[1].op}) belong to stem? {in_stem(conv.args[1])}")
assert in_stem(conv) and not in_stem(conv.args[1])Does conv belong to stem? True
Does the conv's weight (get_attr) belong to stem? False
So on torch 2.14 + torchao 0.18, this XNNPACKQuantizer’s module-level setting has no effect on conv (it works for linear, and the executorch version is the same, M08-1); set_operator_type isn’t read either. After using any config API, print the annotations or observers to confirm.
Summary
- A backend’s
set_*config APIs wrap “rules for finding nodes” for users; they are not part of theQuantizerbase class and differ by backend. X86InductorQuantizer: module name > op type > global, andNonemeans not quantized.- A setting can silently have no effect: after setting it, check the observers or annotations.