Skip to main content

M03-7 | A Backend’s Config API: global, Op Type, Module Name

Open In Colab

In M03-6 we wrote the rules ourselves (node.target + module name filter) to decide which nodes to quantize. Real backends wrap such rules in a config API for users to call. Take X86InductorQuantizer (torchao’s Quantizer for x86 CPU + Inductor) as an example:

Call Applies to
set_global(cfg) Nodes not hit by any other rule
set_module_type_qconfig(nn.Conv2d, cfg) Ops produced by a certain module class
set_module_name_qconfig("stem", cfg) Ops produced by the submodule of a certain name

A cfg of None means this part is not quantized. These set_* methods are not part of the Quantizer base class API; each backend defines its own, and names and behavior vary by backend:

import copy
import dataclasses
from collections import OrderedDict

import torch

from common.data import calibration_batches
from common.graph_utils import observer_nodes, qdq_nodes
from common.models import ConvBnReLU, example_inputs
from common.pt2e import (
    X86InductorQuantizer,
    convert_pt2e,
    export_module,
    get_default_x86_inductor_quantization_config,
    get_module_name_filter,
    get_xnnpack_api,
    prepare_pt2e,
    quiet,
)
from common.viz import table

quiet()
_ = torch.manual_seed(0)
CONV2D = torch.ops.aten.conv2d.default

The model is three conv + bn + relu segments in series, with submodule names stem, block, head_conv:

chain = torch.nn.Sequential(OrderedDict(stem=ConvBnReLU(3, 8), block=ConvBnReLU(8, 8), head_conv=ConvBnReLU(8, 8)))
gm = export_module(chain.eval(), example_inputs(4))
convs = [n.name for n in gm.graph.nodes if n.target == CONV2D]
print("convs in the graph:", convs)
convs in the graph: ['conv2d', 'conv2d_1', 'conv2d_2']

A. X86InductorQuantizer: Whose Priority Is Higher

To know which setting each conv actually got, make a fingerprint for each setting: copy the default config and change only the activation’s quant_max to a different value (global 255, op type 200, module name 100). After prepare, read the quant_max of the observer on the conv input to see which one took effect. These configs are only for telling them apart, so precision isn’t compared here.

g = get_default_x86_inductor_quantization_config()


def with_qmax(qmax):
    """Copy the default config, changing only the activation's quant_max."""
    act = g.input_activation
    assert act is not None
    return dataclasses.replace(g, input_activation=dataclasses.replace(act, quant_max=qmax))


by_type, by_name = with_qmax(200), with_qmax(100)
fingerprint = {255: "global", 200: "op type", 100: "module name"}


def make(use_global=True, type_cfg=None, names=None):
    q = X86InductorQuantizer()
    if use_global:
        q.set_global(g)
    if type_cfg:
        q.set_module_type_qconfig(torch.nn.Conv2d, type_cfg)
    for name, cfg in (names or {}).items():
        q.set_module_name_qconfig(name, cfg)
    return q


def applied(prepared):
    """The setting each conv got: look at the quant_max of its input observer; no observer means fp32."""
    result = []
    for conv in (n for n in prepared.graph.nodes if n.target == CONV2D):
        src = conv.args[0]
        result.append(fingerprint[getattr(prepared.get_submodule(src.target), "quant_max")]
                      if src.op == "call_module" else "fp32")
    return result


settings = [
    ("set_global only", make()),
    ("global + op type (nn.Conv2d)", make(type_cfg=by_type)),
    ("global + op type + module name (stem)", make(type_cfg=by_type, names={"stem": by_name})),
    ("global + module name (block) = None", make(names={"block": None})),
    ("no global, module name (stem) only", make(use_global=False, names={"stem": by_name})),
]
rows = []
for title, quantizer in settings:
    prepared = prepare_pt2e(copy.deepcopy(gm), quantizer)
    used = applied(prepared)
    with torch.no_grad():
        for batch in calibration_batches(n=16, batch_size=16):
            prepared(*batch)
    rows.append([title, *used, len(qdq_nodes(convert_pt2e(prepared)))])
table(rows, ["setting", "stem", "block", "head_conv", "# q/dq"])
setting stem block head_conv # q/dq
set_global only global global global 9
global + op type (nn.Conv2d) op type op type op type 9
global + op type + module name (stem) module name op type op type 9
global + module name (block) = None global fp32 global 6
no global, module name (stem) only module name fp32 fp32 3
assert [r[1:4] for r in rows] == [
    ["global"] * 3,
    ["op type"] * 3,
    ["module name", "op type", "op type"],
    ["global", "fp32", "global"],
    ["module name", "fp32", "fp32"],
]
  • Priority: module name > op type > global (rows 2 and 3). A node is annotated only once, by the first rule that hits it, and later stages skip it when they see it is already annotated (see the description of X86InductorQuantizer.annotate); it is the same mechanism as _annotated in M03-6.
  • Setting None explicitly leaves this section unquantized (row 4); this is the most direct way to write partial quantization.
  • Without a global, nodes not hit by any rule are not quantized (row 5).

Method names differ by backend; for example X86/Arm Inductor use set_module_name_qconfig, while XNNPACK uses set_module_name. The rules we wrote ourselves in M03-6 (node.target + module name filter) are what’s behind these APIs.

B. XNNPACKQuantizer’s set_module_name Has No Effect on conv

Do the same thing with the torchao test-version XNNPACKQuantizer: set_module_name("stem", cfg), expecting stem to be quantized.

xnnpack_cls, config_fn, _ = get_xnnpack_api()
prepared = prepare_pt2e(copy.deepcopy(gm), xnnpack_cls().set_module_name("stem", config_fn(is_per_channel=True)))
print("Number of observers after prepare:", len(observer_nodes(prepared)))
assert len(observer_nodes(prepared)) == 0
Number of observers after prepare: 0

Not a single observer: stem silently stays in fp32, with no error. The cause is in the filter function: it requires every node in the pattern to belong to stem. The conv itself has an nn_module_stack, but the conv’s weight is a get_attr node, which has no such field:

in_stem = get_module_name_filter("stem")
conv = next(n for n in gm.graph.nodes if n.target == CONV2D)
print(f"Does conv belong to stem? {in_stem(conv)}")
print(f"Does the conv's weight ({conv.args[1].op}) belong to stem? {in_stem(conv.args[1])}")
assert in_stem(conv) and not in_stem(conv.args[1])
Does conv belong to stem? True
Does the conv's weight (get_attr) belong to stem? False

So on torch 2.14 + torchao 0.18, this XNNPACKQuantizer’s module-level setting has no effect on conv (it works for linear, and the executorch version is the same, M08-1); set_operator_type isn’t read either. After using any config API, print the annotations or observers to confirm.

Summary

  • A backend’s set_* config APIs wrap “rules for finding nodes” for users; they are not part of the Quantizer base class and differ by backend.
  • X86InductorQuantizer: module name > op type > global, and None means not quantized.
  • A setting can silently have no effect: after setting it, check the observers or annotations.