M02 Overview | Splitting Prepare and Convert Apart: Where Observers Come From and What They Become
M01 used prepare_pt2e and convert_pt2e as black boxes: one inserts observers, the other replaces observers with q/dq. This chapter opens both boxes and redoes them by hand, without calling them, to show that only a few graph-editing actions are inside. The next three notebooks (M02-1 to M02-3) verify each piece hands-on.
This chapter covers only the observer itself and the editing mechanics; where observers go is decided by the Quantizer, which is the topic of the next chapter, M03.
Why Split Them Apart
Writing your own quantizer (M03), understanding how BN gets folded (M04), and arranging the QAT training flow (M06) all require knowing what prepare and convert do to the graph. Otherwise, when something goes wrong you can only guess: what is the observer actually computing? How does it appear in the graph? Where does the scale after convert come from?
This chapter answers three questions:
| Question | One-line answer | Verified hands-on in |
|---|---|---|
| What does an observer compute? | It records the numeric range and derives scale and zero point; different observers pick the range differently | M02-1 |
| How does an observer get into the graph? | Attach a submodule, insert a call_module node, and rewire the downstream |
M02-2 |
| How does an observer become q/dq? | Take out the qparams, replace it with quantize and dequantize nodes, then precompute the int8 weight | M02-3 |
An Example First: a Fully Hand-Made conv → relu
The same conv → relu as in M01, and the three stages look the same. The difference: stages two and three here do not call prepare_pt2e or convert_pt2e; they are made by hand with the graph-editing code from M02-2 and M02-3 (the code is hidden here, only the results are shown). The first row is the export and the hand-inserted observers, the second row is the hand-replaced q/dq:
- Hand-inserted observers: one observer each on the input
x, the conv weight, and the relu output (green boxes), with kinds following the XNNPACKQuantizer configuration. - Hand-replaced q/dq: after calibration, each observer is replaced by a quantize → dequantize pair, with scale and zero point written on the node.
Matching graphs are not enough; the numbers must match too. Hand the same exported graph to prepare_pt2e, calibrate, run convert_pt2e, and compare with the hand-made result:
api_prepared = prepare_pt2e(copy.deepcopy(gm), quantizer)
calibrate(api_prepared)
api_converted = convert_pt2e(copy.deepcopy(api_prepared))
x = torch.randn(4, 3, 16, 16)
with torch.no_grad():
diff = float((api_converted(x) - hand_converted(x)).abs().max())
same_qparams = all(
torch.equal(a.calculate_qparams()[0], b.calculate_qparams()[0])
for (_, a), (_, b) in zip(observer_nodes(api_prepared), observer_nodes(hand_prepared))
)
print(f"Hand-inserted observers have the same scales as prepare_pt2e? {same_qparams}")
print(f"Max difference between hand-replaced q/dq and convert_pt2e outputs:{diff}")
assert same_qparams and diff == 0Hand-inserted observers have the same scales as prepare_pt2e? True
Max difference between hand-replaced q/dq and convert_pt2e outputs:0.0
Exactly identical. The only difference is a graph detail: convert_pt2e also precomputes the weight’s quantize and stores only int8 (part D of M02-3 adds this step).
Definitions
Observer: two methods
An observer is an nn.Module; only two methods matter:
forward(x): record the values ofx(min/max, or a histogram), then returnxunchanged. So a graph with observers inserted produces outputs bit-for-bit identical to the original.calculate_qparams(): derive the scale \(s\) and zero point \(z\) from the recorded range. The conversion formula is in the M01 overview.
Observers differ only in “how the range is chosen”. The ones used in this chapter:
| observer | How the range is chosen |
|---|---|
MinMaxObserver |
The smallest and largest values seen; a single outlier widens the range permanently |
MovingAverageMinMaxObserver |
A moving average of each batch’s min/max, so old outliers fade (Eq. 1) |
PerChannelMinMaxObserver |
Each channel’s own min/max; the scale is a vector |
HistogramObserver |
Records a histogram and searches for a range that lowers the quantization error; in this project, the default XNNPACKQuantizer configuration uses it for activations in PTQ (a configuration choice, not a rule) |
PlaceholderObserver |
Records nothing, only marks the dtype of this edge; dynamic quantization uses it |
The moving-average update rule (\(c\) is averaging_constant; the first batch takes that batch’s value directly); \(x_{\max}\) is analogous:
\[ x_{\min} \leftarrow x_{\min} + c\,\big(x_{\min}^{\text{batch}} - x_{\min}\big) \tag{1} \]
obs = MovingAverageMinMaxObserver(averaging_constant=0.1, dtype=torch.int8)
batches = [torch.randn(1000) * a for a in (1.0, 10.0, 1.0)]
hand = batches[0].min()
for b in batches:
obs(b)
for b in batches[1:]:
hand = hand + 0.1 * (b.min() - hand) # Eq. (1)
print(f"observer min_val = {float(obs.min_val):.4f}; Eq. (1) by hand = {float(hand):.4f}")
assert torch.allclose(obs.min_val, hand)observer min_val = -6.0902; Eq. (1) by hand = -6.0902
Prepare Inserts Observers: Who Decides the Positions
What prepare_pt2e does to insert an observer is the graph editing from M00-4: attach an observer submodule, insert a call_module node, and make the downstream consume it. Which positions and which kind of observer are decided by the annotations that the Quantizer passed in leaves on the graph; prepare_pt2e itself does not know where to quantize. In this chapter, hand insertion follows the positions and settings of XNNPACKQuantizer directly; how annotations are written and how a Quantizer decides positions is the topic of M03.
FakeQuantize: an Observer Plus Quantize → Dequantize
The prepare for QAT (prepare_qat_pt2e) inserts not observers but fake quant modules. A FakeQuantize wraps an observer, and its forward does two things, each with a switch:
| Switch | What it does when on |
|---|---|
observer_enabled |
Use the inner observer to record the range and update scale and zero point |
fake_quant_enabled |
Use the current scale and zero point to do quantize → dequantize, outputting floats that carry quantization error |
An observer only peeks and never changes values; fake quant lets later layers see the quantization error, so QAT can learn to tolerate it. For why QAT can backpropagate through round, see the fake quantization article in the references.
Steps of convert_pt2e
According to the torchao 0.18 source, convert_pt2e does the following in order:
| Order | Step | What it does |
|---|---|---|
| 1 | observer → q/dq | Take out \(s\) and \(z\) with calculate_qparams(), and replace each observer with quantize and dequantize nodes |
| 2 | _fold_conv_bn_qat |
Fold away the conv + bn left in place during QAT training (M04, M06) |
| 3 | DuplicateDQPass |
When one dequantize has multiple users, make a copy for each user so the backend can rewrite them one by one |
| 4 | PortNodeMetaForQDQ |
Move metadata such as annotations onto the q/dq nodes |
| 5 | constant_fold |
The weight’s quantize input is a constant; precompute it and store only int8 |
Because step 1 only calls calculate_qparams(), and FakeQuantize also has this method, the QAT graph takes the same path. M02-3 does steps 1, 3, and 5 by hand, and the result has exactly the same op composition and outputs as convert_pt2e.
Summary and Chapter Map
- Prepare and convert are both just graph edits: prepare inserts
call_module(the observer), and convert replaces it withcall_function(q/dq) and folds away constants. - There are many kinds of observer, differing only in “how the range is chosen”; which kind and where it goes is decided by the Quantizer (M03).
- An observer only records and never changes values; fake quant does change values; the two behaviors have separate switches.
| notebook | Question it answers |
|---|---|
| M02-1 Observer Zoo | symmetric vs. affine, how each observer chooses a range, ch_axis for per-channel, which one real quantizers use |
| M02-2 Insert Observers and Fake Quant by Hand | Insert observers without a Quantizer, replace them with FakeQuantize, the two switches |
| M02-3 Do convert_pt2e by Hand | observer → q/dq, DuplicateDQPass, constant folding of the weight |
What this chapter deliberately skips: who decides observer positions and how to write your own Quantizer (M03); how BN is folded into conv (M04); when QAT flips the switches (M06); how a Q/DQ graph becomes integer kernels (M07 onward).
References
- torchao 0.18 source:
quantize_pt2e.py(convert steps),convert.py(observer → q/dq),observer.py,fake_quantize.py,duplicate_dq_pass.py - Official PyTorch tutorials: PyTorch 2 Export Post Training Quantization, How to Write a Quantizer for PyTorch 2 Export Quantization
- Some Basic Concepts of PyTorch FX Graph / Node (in Chinese): the node and graph operations used for graph editing
- Understanding Fake Quantization in Quantization Aware Training (in Chinese): why fake quant can backpropagate
- Quantization Error (Case without Clipping), Quantization Error (Case with Clipping): quantization error splits into a discretization term and a clipping term, and choosing a larger or smaller range is a trade-off between them; this is the way to understand why M02-1’s
HistogramObserverdoes not simply use min/max