import copy
import torch
from torch.fx import Node
from common.graph_utils import observer_nodes
from common.models import example_inputs, get_model
from common.pt2e import (
ANNOTATION_KEY,
MovingAverageMinMaxObserver,
QuantizationSpec,
Quantizer,
SharedQuantizationSpec,
annotate_input_qspec_map,
annotate_output_qspec,
convert_pt2e,
export_module,
prepare_pt2e,
quiet,
)
from common.qspecs import act_spec
from common.viz import draw, table
quiet()
_ = torch.manual_seed(0)
ADD = torch.ops.aten.add.TensorM03-4 | Shared qparams: SharedQuantizationSpec and Implicit Sharing
To do integer addition (or concat) on two int8 tensors, they must be on the same scale: integers with different scales can’t be added directly, and the hardware would have to rescale one of them first. So some backends require all inputs and outputs of add and cat to share one set of (scale, zero point).
In PT2E, “sharing qparams” means making several positions share the same observer instance: this observer sees the values at all the positions, computes one scale, and everyone uses it. Sharing comes from two sources:
- Explicit: the annotation’s spec is
SharedQuantizationSpec(some position), meaning “use the same observer as that position”. - Implicit: when two annotations on the same tensor have identical attributes,
prepare_pt2emerges them into one observer automatically. Mentioned in M03-1 section C.
Sections:
- A. Explicit sharing: make the inputs and output of add share one set of qparams.
- B. Implicit sharing: when the same tensor has both an output annotation and an edge annotation, when they merge and when they chain.
B. Implicit Sharing: Output Annotation + Edge Annotation
M03-1 section C showed that one tensor can have an output annotation (affecting all consumers) and also an edge annotation (affecting only one consumer). What happens when both are attached?
Use the cat_pool model (the input x feeds both conv2d and conv2d_1). TwoLabelQuantizer attaches spec A to the output of x, and spec B to the edge x → conv2d_1. It is written for this experiment by naming nodes, so the node names only fit cat_pool. allow_implicit_sharing is a field of QuantizationAnnotation (default True); setting it to False turns merging off.
class TwoLabelQuantizer(Quantizer):
def __init__(self, spec_a, spec_b, implicit=True):
self.spec_a, self.spec_b, self.implicit = spec_a, spec_b, implicit
def annotate(self, model):
nodes = {n.name: n for n in model.graph.nodes}
annotate_output_qspec(nodes["x"], self.spec_a)
annotate_input_qspec_map(nodes["conv2d_1"], nodes["x"], self.spec_b)
nodes["conv2d_1"].meta[ANNOTATION_KEY].allow_implicit_sharing = self.implicit
return model
def validate(self, model):
passSpec B in the four cases:
a = act_spec() # int8 per-tensor affine [-128, 127], MinMaxObserver
different = QuantizationSpec( # qscheme and integer range both differ from A
dtype=torch.int8, quant_min=-127, quant_max=127, qscheme=torch.per_tensor_symmetric,
observer_or_fake_quant_ctr=a.observer_or_fake_quant_ctr,
)
other_observer = QuantizationSpec( # quantization attributes identical to A; only the observer class differs
dtype=a.dtype, quant_min=a.quant_min, quant_max=a.quant_max, qscheme=a.qscheme,
observer_or_fake_quant_ctr=MovingAverageMinMaxObserver,
)
cases = [
("B differs from A in attributes", TwoLabelQuantizer(a, different)),
("B has the same attributes as A", TwoLabelQuantizer(a, a)),
("same attributes, but allow_implicit_sharing=False", TwoLabelQuantizer(a, a, implicit=False)),
("B differs from A only in observer class", TwoLabelQuantizer(a, other_observer)),
]For each case, prepare, calibrate and convert, then print what x passes through on its way to conv2d_1 (obs = observer, q/dq = quantize/dequantize):
def path_to(gm, consumer):
"""The nodes x passes through, in order, on its way to the consumer."""
steps, src = [], next(n for n in gm.graph.nodes if n.name == consumer).args[0]
while src.op != "placeholder":
target = str(src.target)
steps.append("obs" if src.op == "call_module" else "dq" if "dequantize" in target else "q")
src = src.args[0]
return " → ".join(["x", *reversed(steps), consumer])
gm_cat = export_module(get_model("cat_pool").eval(), example_inputs(4))
rows = []
for title, quantizer in cases:
prepared = prepare_pt2e(copy.deepcopy(gm_cat), quantizer)
obs = observer_nodes(prepared)
kinds = ", ".join(sorted({type(m).__name__ for _, m in obs}))
before = path_to(prepared, "conv2d_1")
with torch.no_grad():
for _ in range(4):
prepared(torch.randn(4, 3, 16, 16))
rows.append([title, len(obs), kinds, before, path_to(convert_pt2e(prepared), "conv2d_1")])
table(rows, ["case", "# observers", "observer classes", "after prepare", "after convert"])| case | # observers | observer classes | after prepare | after convert |
|---|---|---|---|---|
| B differs from A in attributes | 2 | MinMaxObserver | x → obs → obs → conv2d_1 | x → q → dq → q → dq → conv2d_1 |
| B has the same attributes as A | 1 | MinMaxObserver | x → obs → conv2d_1 | x → q → dq → conv2d_1 |
| same attributes, but allow_implicit_sharing=False | 2 | MinMaxObserver | x → obs → obs → conv2d_1 | x → q → dq → q → dq → conv2d_1 |
| B differs from A only in observer class | 1 | MinMaxObserver | x → obs → conv2d_1 | x → q → dq → conv2d_1 |
assert [r[1] for r in rows] == [2, 1, 2, 1]
assert rows[0][4] == "x → q → dq → q → dq → conv2d_1" and rows[1][4] == "x → q → dq → conv2d_1"
assert rows[3][2] == "MinMaxObserver"- Different attributes: the two annotations chain.
xis first quantized by A, then quantized again by B; after convert there are two layers of q/dq, rounding twice. Neither overrides the other, and no error is raised. - Identical attributes: automatically merged into one observer, quantized only once. They are the same tensor anyway, so there is no need to quantize twice; this is implicit sharing.
allow_implicit_sharing=False: no merging, back to two chained observers. Once it is off, you are responsible for not quantizing the same tensor twice.- What is compared is the quantization attributes (dtype, integer range, qscheme, …), not the observer class. When only the observer class differs, they also merge, and what remains is A’s
MinMaxObserver; theMovingAverageMinMaxObserverspecified by B is silently ignored (in this graph A’s annotation comes first in graph order).
To make the same tensor quantized only once: make the attributes of all annotations consistent, or explicitly point to the same position with SharedQuantizationSpec.
Summary
- Sharing qparams = several positions share the same observer instance; the graph still has several observer nodes.
- Explicit:
SharedQuantizationSpec(position), where the position is an edge(producer, consumer)or a node’s output. - Implicit: annotations with identical attributes on the same tensor merge automatically; with different attributes they chain into two quantizations, without an error.