Skip to main content

M06-2 | Two Switches to Flip Before Evaluating a QAT Model: BN train/eval, and the Observers

Open In Colab

M05-2 showed that the PTQ graph has no BN, so calibration is not affected by batch composition. The QAT graph is different:

Both make the output of “the same sample” change with “who it runs together with” and “what ran before”.

import copy

import torch

from common.data import calibration_batches, evaluate_loss, make_loader, trained_tiny_net
from common.models import example_inputs
from common.pt2e import (
    allow_exported_model_train_eval,
    disable_observer,
    enable_observer,
    export_module,
    move_exported_model_to_eval,
    move_exported_model_to_train,
    prepare_qat_pt2e,
    quiet,
)
from common.qat import qat_quantizer, train_qat
from common.viz import table

quiet()
_ = torch.manual_seed(0)

First train an int8 QAT model. train_qat() is this project’s tool (common/qat.py); its body is the loop written out line by line in M06-1 B, plus a hook called at every step. When it finishes, the model is in eval mode with the observers still on.

model = trained_tiny_net()
test_loader = make_loader(n=1024, seed=2, shuffle=False)
(x16,) = next(calibration_batches(16, seed=5))

prepared = prepare_qat_pt2e(export_module(model, example_inputs(), train=True), qat_quantizer(bits=8))
_ = train_qat(prepared, epochs=3)

A. Three States

Two switches:

  • BN: move_exported_model_to_train/move_exported_model_to_eval (M06-1).
  • Observer: model.apply(enable_observer)/model.apply(disable_observer), which flips each fake quant’s observer_enabled buffer (M02-2). The other switch, fake_quant_enabled, is left alone here.

For each state we measure two things:

  • Alone vs together: the max difference of the output between running 16 samples one at a time and running them all at once.
  • Evaluate twice: whether the test loss is the same when the same test set is evaluated twice in a row.

Each run is on a deepcopy so that side effects of one do not affect the next.

@torch.no_grad()
def alone_vs_together(gm, x):
    together = copy.deepcopy(gm)(x)
    g = copy.deepcopy(gm)
    alone = torch.cat([g(x[i : i + 1]) for i in range(len(x))])
    return float((alone - together).abs().max())


STATES = {
    "train mode, observer on": (move_exported_model_to_train, enable_observer),
    "eval mode, observer on": (move_exported_model_to_eval, enable_observer),
    "eval mode, observer off": (move_exported_model_to_eval, disable_observer),
}
results = {}
for name, (set_bn, set_observer) in STATES.items():
    g = copy.deepcopy(prepared)
    set_bn(g)
    g.apply(set_observer)
    diff = alone_vs_together(g, x16)
    results[name] = (diff, evaluate_loss(g, test_loader), evaluate_loss(g, test_loader))
table([[name, f"{d:.1e}", f"{l1:.5f}", f"{l2:.5f}", "yes" if l1 == l2 else "no"] for name, (d, l1, l2) in results.items()],
      ["State", "Alone vs together: max difference", "test loss, 1st time", "2nd time", "Reproducible"])
State Alone vs together: max difference test loss, 1st time 2nd time Reproducible
train mode, observer on 6.2e+00 0.88114 0.87998 no
eval mode, observer on 2.6e-01 0.82947 0.82836 no
eval mode, observer off 0.0e+00 0.82784 0.82784 yes
reproducible = [l1 == l2 for _, l1, l2 in results.values()]
assert reproducible == [False, False, True]
assert results["train mode, observer on"][0] > results["eval mode, observer on"][0] > 0
  • train mode: BN uses batch statistics, so the output of the same sample changes with the other samples in the batch. Evaluation itself also updates the running stats, so even two evaluations in a row differ.
  • eval mode, observer on: BN is fixed, but every forward keeps updating the fake quant’s min/max (moving average), so scale moves, and the output moves with it. The evaluation data quietly takes part in calibration.
  • eval mode, observer off: the two evaluations are exactly identical; this is the number you can report.

In the third state, alone vs together can still occasionally differ by “exactly one grid step”: on CPU, conv at batch 1 and batch 16 takes different implementations, with a floating-point difference of about 1e-5 that happens to push a value sitting on a round boundary into the neighboring step. This is unrelated to the quantization flow; see M07-1.

So before evaluation or convert, always do two things: move_exported_model_to_eval and apply(disable_observer). This project wraps these two lines as freeze_for_eval() (common/qat.py), and later notebooks all use it. The same applies when you evaluate midway through training: after evaluating, enable_observer again and switch back to train.

B. allow_exported_model_train_eval

If you do not want to write move_exported_model_to_train every time, you can use allow_exported_model_train_eval(gm) to replace the GraphModule’s .train()/.eval() with these two functions. Below we read the training argument (the 6th positional argument) of the 4 batch_norm nodes in the graph:

def bn_training_flags(gm):
    return [n.args[5] for n in gm.graph.nodes if n.op == "call_function" and "batch_norm" in str(n.target)]


g = copy.deepcopy(prepared)
flags = {"(after training)": bn_training_flags(g)}
allow_exported_model_train_eval(g)
g.train()
flags[".train()"] = bn_training_flags(g)
g.eval()
flags[".eval()"] = bn_training_flags(g)
table([[k, str(v)] for k, v in flags.items()], ["Call", "training argument of batch_norm"])
Call training argument of batch_norm
(after training) [False, False, False, False]
.train() [True, True, True, True]
.eval() [False, False, False, False]
assert [set(v) for v in flags.values()] == [{False}, {True}, {False}]
  • It only replaces .train()/.eval() with move_exported_model_to_train/eval (from reading the torchao 0.18 source), and still affects only BN and dropout; observers are not affected.
  • The model.training attribute has different semantics from an ordinary nn.Module; do not use it to judge the state.
  • The benefit is that training/evaluation functions “written for an ordinary nn.Module” can be used directly on the QAT graph. This chapter writes move_exported_model_to_train/eval explicitly so that the timing of the switch is visible.

Summary

  • Before evaluation or convert: switch BN to eval and turn the observers off (freeze_for_eval). Skip either one and the result is not reproducible.
  • allow_exported_model_train_eval makes .train()/.eval() usable, but still does not handle observers.