import copy
import torch
from common.data import calibration_batches, evaluate_loss, make_loader, trained_tiny_net
from common.models import example_inputs
from common.pt2e import (
allow_exported_model_train_eval,
disable_observer,
enable_observer,
export_module,
move_exported_model_to_eval,
move_exported_model_to_train,
prepare_qat_pt2e,
quiet,
)
from common.qat import qat_quantizer, train_qat
from common.viz import table
quiet()
_ = torch.manual_seed(0)M06-2 | Two Switches to Flip Before Evaluating a QAT Model: BN train/eval, and the Observers
M05-2 showed that the PTQ graph has no BN, so calibration is not affected by batch composition. The QAT graph is different:
- BN is still there (M04-2), and in train mode it uses the statistics of the current batch.
- As long as the observer inside a fake quant is on, every forward updates min/max, and scale changes with it.
Both make the output of “the same sample” change with “who it runs together with” and “what ran before”.
- A. In three states, how much the same sample differs when run alone versus in the same batch, and whether the loss is the same when evaluating the same test set twice.
- B.
allow_exported_model_train_eval: wire.train()/.eval()back onto the GraphModule.
First train an int8 QAT model. train_qat() is this project’s tool (common/qat.py); its body is the loop written out line by line in M06-1 B, plus a hook called at every step. When it finishes, the model is in eval mode with the observers still on.
model = trained_tiny_net()
test_loader = make_loader(n=1024, seed=2, shuffle=False)
(x16,) = next(calibration_batches(16, seed=5))
prepared = prepare_qat_pt2e(export_module(model, example_inputs(), train=True), qat_quantizer(bits=8))
_ = train_qat(prepared, epochs=3)A. Three States
Two switches:
- BN:
move_exported_model_to_train/move_exported_model_to_eval(M06-1). - Observer:
model.apply(enable_observer)/model.apply(disable_observer), which flips each fake quant’sobserver_enabledbuffer (M02-2). The other switch,fake_quant_enabled, is left alone here.
For each state we measure two things:
- Alone vs together: the max difference of the output between running 16 samples one at a time and running them all at once.
- Evaluate twice: whether the test loss is the same when the same test set is evaluated twice in a row.
Each run is on a deepcopy so that side effects of one do not affect the next.
@torch.no_grad()
def alone_vs_together(gm, x):
together = copy.deepcopy(gm)(x)
g = copy.deepcopy(gm)
alone = torch.cat([g(x[i : i + 1]) for i in range(len(x))])
return float((alone - together).abs().max())
STATES = {
"train mode, observer on": (move_exported_model_to_train, enable_observer),
"eval mode, observer on": (move_exported_model_to_eval, enable_observer),
"eval mode, observer off": (move_exported_model_to_eval, disable_observer),
}
results = {}
for name, (set_bn, set_observer) in STATES.items():
g = copy.deepcopy(prepared)
set_bn(g)
g.apply(set_observer)
diff = alone_vs_together(g, x16)
results[name] = (diff, evaluate_loss(g, test_loader), evaluate_loss(g, test_loader))
table([[name, f"{d:.1e}", f"{l1:.5f}", f"{l2:.5f}", "yes" if l1 == l2 else "no"] for name, (d, l1, l2) in results.items()],
["State", "Alone vs together: max difference", "test loss, 1st time", "2nd time", "Reproducible"])| State | Alone vs together: max difference | test loss, 1st time | 2nd time | Reproducible |
|---|---|---|---|---|
| train mode, observer on | 6.2e+00 | 0.88114 | 0.87998 | no |
| eval mode, observer on | 2.6e-01 | 0.82947 | 0.82836 | no |
| eval mode, observer off | 0.0e+00 | 0.82784 | 0.82784 | yes |
reproducible = [l1 == l2 for _, l1, l2 in results.values()]
assert reproducible == [False, False, True]
assert results["train mode, observer on"][0] > results["eval mode, observer on"][0] > 0- train mode: BN uses batch statistics, so the output of the same sample changes with the other samples in the batch. Evaluation itself also updates the running stats, so even two evaluations in a row differ.
- eval mode, observer on: BN is fixed, but every forward keeps updating the fake quant’s min/max (moving average), so scale moves, and the output moves with it. The evaluation data quietly takes part in calibration.
- eval mode, observer off: the two evaluations are exactly identical; this is the number you can report.
In the third state, alone vs together can still occasionally differ by “exactly one grid step”: on CPU, conv at batch 1 and batch 16 takes different implementations, with a floating-point difference of about 1e-5 that happens to push a value sitting on a round boundary into the neighboring step. This is unrelated to the quantization flow; see M07-1.
So before evaluation or convert, always do two things: move_exported_model_to_eval and apply(disable_observer). This project wraps these two lines as freeze_for_eval() (common/qat.py), and later notebooks all use it. The same applies when you evaluate midway through training: after evaluating, enable_observer again and switch back to train.
B. allow_exported_model_train_eval
If you do not want to write move_exported_model_to_train every time, you can use allow_exported_model_train_eval(gm) to replace the GraphModule’s .train()/.eval() with these two functions. Below we read the training argument (the 6th positional argument) of the 4 batch_norm nodes in the graph:
def bn_training_flags(gm):
return [n.args[5] for n in gm.graph.nodes if n.op == "call_function" and "batch_norm" in str(n.target)]
g = copy.deepcopy(prepared)
flags = {"(after training)": bn_training_flags(g)}
allow_exported_model_train_eval(g)
g.train()
flags[".train()"] = bn_training_flags(g)
g.eval()
flags[".eval()"] = bn_training_flags(g)
table([[k, str(v)] for k, v in flags.items()], ["Call", "training argument of batch_norm"])| Call | training argument of batch_norm |
|---|---|
| (after training) | [False, False, False, False] |
| .train() | [True, True, True, True] |
| .eval() | [False, False, False, False] |
assert [set(v) for v in flags.values()] == [{False}, {True}, {False}]- It only replaces
.train()/.eval()withmove_exported_model_to_train/eval(from reading the torchao 0.18 source), and still affects only BN and dropout; observers are not affected. - The
model.trainingattribute has different semantics from an ordinarynn.Module; do not use it to judge the state. - The benefit is that training/evaluation functions “written for an ordinary
nn.Module” can be used directly on the QAT graph. This chapter writesmove_exported_model_to_train/evalexplicitly so that the timing of the switch is visible.
Summary
- Before evaluation or convert: switch BN to eval and turn the observers off (
freeze_for_eval). Skip either one and the result is not reproducible. allow_exported_model_train_evalmakes.train()/.eval()usable, but still does not handle observers.