Skip to main content

M02 Overview | Splitting Prepare and Convert Apart: Where Observers Come From and What They Become

Open In Colab

M01 used prepare_pt2e and convert_pt2e as black boxes: one inserts observers, the other replaces observers with q/dq. This chapter opens both boxes and redoes them by hand, without calling them, to show that only a few graph-editing actions are inside. The next three notebooks (M02-1 to M02-3) verify each piece hands-on.

This chapter covers only the observer itself and the editing mechanics; where observers go is decided by the Quantizer, which is the topic of the next chapter, M03.

Why Split Them Apart

Writing your own quantizer (M03), understanding how BN gets folded (M04), and arranging the QAT training flow (M06) all require knowing what prepare and convert do to the graph. Otherwise, when something goes wrong you can only guess: what is the observer actually computing? How does it appear in the graph? Where does the scale after convert come from?

This chapter answers three questions:

Question One-line answer Verified hands-on in
What does an observer compute? It records the numeric range and derives scale and zero point; different observers pick the range differently M02-1
How does an observer get into the graph? Attach a submodule, insert a call_module node, and rewire the downstream M02-2
How does an observer become q/dq? Take out the qparams, replace it with quantize and dequantize nodes, then precompute the int8 weight M02-3

An Example First: a Fully Hand-Made conv → relu

The same conv → relu as in M01, and the three stages look the same. The difference: stages two and three here do not call prepare_pt2e or convert_pt2e; they are made by hand with the graph-editing code from M02-2 and M02-3 (the code is hidden here, only the results are shown). The first row is the export and the hand-inserted observers, the second row is the hand-replaced q/dq:

  • Hand-inserted observers: one observer each on the input x, the conv weight, and the relu output (green boxes), with kinds following the XNNPACKQuantizer configuration.
  • Hand-replaced q/dq: after calibration, each observer is replaced by a quantize → dequantize pair, with scale and zero point written on the node.

Matching graphs are not enough; the numbers must match too. Hand the same exported graph to prepare_pt2e, calibrate, run convert_pt2e, and compare with the hand-made result:

api_prepared = prepare_pt2e(copy.deepcopy(gm), quantizer)
calibrate(api_prepared)
api_converted = convert_pt2e(copy.deepcopy(api_prepared))

x = torch.randn(4, 3, 16, 16)
with torch.no_grad():
    diff = float((api_converted(x) - hand_converted(x)).abs().max())
same_qparams = all(
    torch.equal(a.calculate_qparams()[0], b.calculate_qparams()[0])
    for (_, a), (_, b) in zip(observer_nodes(api_prepared), observer_nodes(hand_prepared))
)
print(f"Hand-inserted observers have the same scales as prepare_pt2e? {same_qparams}")
print(f"Max difference between hand-replaced q/dq and convert_pt2e outputs:{diff}")
assert same_qparams and diff == 0
Hand-inserted observers have the same scales as prepare_pt2e? True
Max difference between hand-replaced q/dq and convert_pt2e outputs:0.0

Exactly identical. The only difference is a graph detail: convert_pt2e also precomputes the weight’s quantize and stores only int8 (part D of M02-3 adds this step).

Definitions

Observer: two methods

An observer is an nn.Module; only two methods matter:

  • forward(x): record the values of x (min/max, or a histogram), then return x unchanged. So a graph with observers inserted produces outputs bit-for-bit identical to the original.
  • calculate_qparams(): derive the scale \(s\) and zero point \(z\) from the recorded range. The conversion formula is in the M01 overview.

Observers differ only in “how the range is chosen”. The ones used in this chapter:

observer How the range is chosen
MinMaxObserver The smallest and largest values seen; a single outlier widens the range permanently
MovingAverageMinMaxObserver A moving average of each batch’s min/max, so old outliers fade (Eq. 1)
PerChannelMinMaxObserver Each channel’s own min/max; the scale is a vector
HistogramObserver Records a histogram and searches for a range that lowers the quantization error; in this project, the default XNNPACKQuantizer configuration uses it for activations in PTQ (a configuration choice, not a rule)
PlaceholderObserver Records nothing, only marks the dtype of this edge; dynamic quantization uses it

The moving-average update rule (\(c\) is averaging_constant; the first batch takes that batch’s value directly); \(x_{\max}\) is analogous:

\[ x_{\min} \leftarrow x_{\min} + c\,\big(x_{\min}^{\text{batch}} - x_{\min}\big) \tag{1} \]

obs = MovingAverageMinMaxObserver(averaging_constant=0.1, dtype=torch.int8)
batches = [torch.randn(1000) * a for a in (1.0, 10.0, 1.0)]
hand = batches[0].min()
for b in batches:
    obs(b)
for b in batches[1:]:
    hand = hand + 0.1 * (b.min() - hand)  # Eq. (1)
print(f"observer min_val = {float(obs.min_val):.4f}; Eq. (1) by hand = {float(hand):.4f}")
assert torch.allclose(obs.min_val, hand)
observer min_val = -6.0902; Eq. (1) by hand = -6.0902

Prepare Inserts Observers: Who Decides the Positions

What prepare_pt2e does to insert an observer is the graph editing from M00-4: attach an observer submodule, insert a call_module node, and make the downstream consume it. Which positions and which kind of observer are decided by the annotations that the Quantizer passed in leaves on the graph; prepare_pt2e itself does not know where to quantize. In this chapter, hand insertion follows the positions and settings of XNNPACKQuantizer directly; how annotations are written and how a Quantizer decides positions is the topic of M03.

FakeQuantize: an Observer Plus Quantize → Dequantize

The prepare for QAT (prepare_qat_pt2e) inserts not observers but fake quant modules. A FakeQuantize wraps an observer, and its forward does two things, each with a switch:

Switch What it does when on
observer_enabled Use the inner observer to record the range and update scale and zero point
fake_quant_enabled Use the current scale and zero point to do quantize → dequantize, outputting floats that carry quantization error

An observer only peeks and never changes values; fake quant lets later layers see the quantization error, so QAT can learn to tolerate it. For why QAT can backpropagate through round, see the fake quantization article in the references.

Steps of convert_pt2e

According to the torchao 0.18 source, convert_pt2e does the following in order:

Order Step What it does
1 observer → q/dq Take out \(s\) and \(z\) with calculate_qparams(), and replace each observer with quantize and dequantize nodes
2 _fold_conv_bn_qat Fold away the conv + bn left in place during QAT training (M04, M06)
3 DuplicateDQPass When one dequantize has multiple users, make a copy for each user so the backend can rewrite them one by one
4 PortNodeMetaForQDQ Move metadata such as annotations onto the q/dq nodes
5 constant_fold The weight’s quantize input is a constant; precompute it and store only int8

Because step 1 only calls calculate_qparams(), and FakeQuantize also has this method, the QAT graph takes the same path. M02-3 does steps 1, 3, and 5 by hand, and the result has exactly the same op composition and outputs as convert_pt2e.

Summary and Chapter Map

  • Prepare and convert are both just graph edits: prepare inserts call_module (the observer), and convert replaces it with call_function (q/dq) and folds away constants.
  • There are many kinds of observer, differing only in “how the range is chosen”; which kind and where it goes is decided by the Quantizer (M03).
  • An observer only records and never changes values; fake quant does change values; the two behaviors have separate switches.
notebook Question it answers
M02-1 Observer Zoo symmetric vs. affine, how each observer chooses a range, ch_axis for per-channel, which one real quantizers use
M02-2 Insert Observers and Fake Quant by Hand Insert observers without a Quantizer, replace them with FakeQuantize, the two switches
M02-3 Do convert_pt2e by Hand observer → q/dq, DuplicateDQPass, constant folding of the weight

What this chapter deliberately skips: who decides observer positions and how to write your own Quantizer (M03); how BN is folded into conv (M04); when QAT flips the switches (M06); how a Q/DQ graph becomes integer kernels (M07 onward).

References