Skip to main content

M07-1 | The fp32 Numeric Floor: Same fp32 Model, Different Kernel, Different Output

Open In Colab

This chapter, together with M08 and M09, compares several implementations of “the same quantized model”: the Q/DQ graph, reference representation, lowering, torch.compile, the backend’s int8 kernel, and ONNX Runtime. Before comparing, we need to know one thing: for an fp32 model that is not quantized at all, how much does the output change just by swapping the kernel? That magnitude is the floor for a “reasonable difference”: a gap smaller than it is not caused by quantization or lowering.

M06-2 showed a phenomenon: with the QAT graph in eval mode and the observers turned off, the same sample run “alone” and run “in a batch with 15 others” occasionally differs by exactly one step. This notebook traces it to its source.

import contextlib

import torch
import torch.nn.functional as F

from common.data import calibration_batches, trained_tiny_net
from common.env import cpu_profile, env_summary
from common.pt2e import quiet
from common.viz import table

quiet()
_ = torch.manual_seed(0)

The numbers depend on the platform: which conv implementation the CPU picks, and at which batch size it switches, depends on the CPU architecture, the PyTorch build (with or without mkldnn), the thread count, and the input size. The outputs stored below were produced in this environment; when you run it on your machine, the notes under the tables are printed according to your results. env_summary() is a tool of this project (common/env.py) that prints a one-line environment summary:

print(env_summary())
Linux x86_64 | torch 2.14.0+cu130 | threads=1 | mkldnn=True | nnpack=True

A. A single conv2d: the output of sample 1 under different batch sizes

Same weights, same sample; only “how many other samples are in the batch” changes. We also compute a reference answer in fp64 to see how far each implementation is from it:

(x,) = next(calibration_batches(32, batch_size=32, seed=5))
w = torch.randn(16, 3, 3, 3)
BATCHES = (1, 2, 8, 15, 16, 32)

with torch.no_grad():
    ref64 = F.conv2d(x.double(), w.double(), padding=1)
    y1 = F.conv2d(x[:1], w, padding=1)
    diffs, errs = {}, {}
    for n in BATCHES:
        yn = F.conv2d(x[:n], w, padding=1)
        diffs[n] = float((yn[:1] - y1).abs().max())
        errs[n] = float((yn.double() - ref64[:n]).abs().max())
table([[n, f"{diffs[n]:.1e}", f"{errs[n]:.1e}"] for n in BATCHES], ["batch", "sample 1 vs batch 1", "max error vs fp64"])
batch sample 1 vs batch 1 max error vs fp64
1 0.0e+00 2.2e-06
2 2.9e-06 3.7e-06
8 2.9e-06 4.5e-06
15 2.9e-06 4.5e-06
16 2.9e-06 4.5e-06
32 2.9e-06 5.2e-06
switch = next((n for n in BATCHES if diffs[n] > 0), None)
if switch:
    print(f"The output of sample 1 is bit-identical up to batch {BATCHES[BATCHES.index(switch) - 1]}, and starts to differ by {diffs[switch]:.1e} at batch {switch}.")
else:
    print(f"The output of sample 1 is bit-identical for batch {BATCHES[0]} to {BATCHES[-1]}: this machine's conv does not switch implementation by batch (section C uses manual switching instead).")
assert max(diffs.values()) < 1e-3  # the difference is on the order of fp32 rounding
The output of sample 1 is bit-identical up to batch 1, and starts to differ by 2.9e-06 at batch 2.

Same sample, same weights; just more samples in the batch, and the result may already differ. Next we check whether the kernel changed.

B. The kernel each batch size actually runs

Use the profiler to record one F.conv2d call and list the ATen kernels actually invoked (dropping the outer conv2d / convolution wrappers). cpu_profile() is a tool of this project (common/env.py): it is just torch.profiler.profile(activities=[CPU]), with the noisy messages the profiler prints at the C level suppressed.

WRAPPERS = ("aten::conv2d", "aten::convolution", "aten::_convolution", "aten::_nnpack_available")


def conv_kernels(x, w):
    with cpu_profile() as prof:
        F.conv2d(x, w, padding=1)
    return ", ".join(e.name for e in prof.events() or [] if e.name.startswith("aten::") and e.name not in WRAPPERS and "conv" in e.name)


kernels = {n: conv_kernels(x[:n], w) for n in BATCHES}
table([[n, kernels[n]] for n in BATCHES], ["batch", "ATen kernel"])
batch ATen kernel
1 aten::thnn_conv2d, aten::_slow_conv2d_forward
2 aten::mkldnn_convolution
8 aten::mkldnn_convolution
15 aten::mkldnn_convolution
16 aten::mkldnn_convolution
32 aten::mkldnn_convolution
groups = {}
for n in BATCHES:
    groups.setdefault(kernels[n], []).append(n)
for k, ns in groups.items():
    print(f"batch {', '.join(map(str, ns))} → {k}")
same_kernel_same_output = all(diffs[n] == 0 for n in BATCHES if kernels[n] == kernels[1])
print("Samples that run the same kernel as batch 1 are bit-identical for sample 1:", same_kernel_same_output)
assert same_kernel_same_output
batch 1 → aten::thnn_conv2d, aten::_slow_conv2d_forward
batch 2, 8, 15, 16, 32 → aten::mkldnn_convolution
Samples that run the same kernel as batch 1 are bit-identical for sample 1: True

The CPU conv picks an implementation by shape: outputs are bit-identical within one implementation, and differ only across implementations (compare section A). The switching threshold depends on the CPU, the PyTorch build, the thread count, and the input size.

C. Switch kernels by hand: same batch, different implementation

torch.backends.nnpack.flags(enabled=False) and torch.backends.mkldnn.flags(enabled=False) are public PyTorch switches; once off, conv falls back to the generic implementation. This lets you see the “kernel-only” difference on any machine:

CONFIGS = (("default", False, False), ("NNPACK off", True, False), ("mkldnn off", False, True), ("both off", True, True))


def fast_paths(nnpack_off=False, mkldnn_off=False):
    """Temporarily turn off the CPU conv fast paths; they are restored on leaving the with block."""
    stack = contextlib.ExitStack()
    if nnpack_off:
        stack.enter_context(torch.backends.nnpack.flags(enabled=False))
    if mkldnn_off:
        stack.enter_context(torch.backends.mkldnn.flags(enabled=False))
    return stack


outs, used = {}, {}
with torch.no_grad():
    for name, nn_off, mk_off in CONFIGS:
        with fast_paths(nn_off, mk_off):
            outs[name] = F.conv2d(x, w, padding=1)
            used[name] = conv_kernels(x, w)
table([[name, used[name], f"{float((outs[name] - outs['default']).abs().max()):.1e}",
        f"{float((outs[name].double() - ref64).abs().max()):.1e}"] for name, _, _ in CONFIGS],
      ["setting", "ATen kernel", "vs default", "vs fp64"])
setting ATen kernel vs default vs fp64
default aten::mkldnn_convolution 0.0e+00 5.2e-06
NNPACK off aten::mkldnn_convolution 0.0e+00 5.2e-06
mkldnn off aten::_nnpack_spatial_convolution 1.1e-04 1.1e-04
both off aten::thnn_conv2d, aten::_slow_conv2d_forward 3.8e-06 4.1e-06
c_max = max(float((o - outs["default"]).abs().max()) for o in outs.values())
print(f"Max difference after switching implementation: {c_max:.1e}" if c_max > 0 else "This machine has no fast path to turn off; all four settings use the same implementation.")
assert all(torch.equal(outs[a], outs[b]) for a in used for b in used if used[a] == used[b])  # same kernel -> bit-identical
assert c_max < 1e-3
Max difference after switching implementation: 1.1e-04
  • With the same input and the same weights, swapping only the kernel changes the output; configurations that run the same kernel are bit-identical.
  • A fast implementation can have a larger error against fp64. NNPACK is farther from fp64 than the generic implementation; our inference is that it uses a fast algorithm such as Winograd / FFT (we did not read the source to confirm).

D. The whole fp32 model: run alone vs. run in a batch

Use M05’s TinyConvNet (trained_tiny_net()), running 16 samples one by one vs. all at once in a batch. We measure under the default setting and under “both off”, and separately measure the final linear:

model, x16 = trained_tiny_net(), x[:16]
d = {}
with torch.no_grad():
    for name, nn_off, mk_off in (CONFIGS[0], CONFIGS[-1]):
        with fast_paths(nn_off, mk_off):
            alone = torch.cat([model(x16[i : i + 1]) for i in range(16)])
            d[f"whole model, {name}"] = float((alone - model(x16)).abs().max())
    h = torch.randn(16, model.fc.in_features)  # look at the final linear only: an input with the same 16 rows
    d["fc (linear) only"] = float((torch.cat([model.fc(h[i : i + 1]) for i in range(16)]) - model.fc(h)).abs().max())
table([[k, f"{v:.1e}"] for k, v in d.items()], ["", "max diff, 16 samples alone vs in a batch"])
max diff, 16 samples alone vs in a batch
whole model, default 3.8e-06
whole model, both off 9.5e-07
fc (linear) only 9.5e-07
d_default, d_off, d_fc = d.values()
if d_default > 0 and d_off == 0:
    print("The difference disappears after turning off the conv fast path: the \"alone vs. in a batch\" gap comes entirely from conv switching implementation.")
elif d_off > 0 and d_fc > 0:
    print(f"{d_off:.1e} remains after turning off the conv fast path; the linear alone differs by {d_fc:.1e} between batch 1 and 16.")
elif d_default > 0:
    print("A difference remains after turning off the conv fast path: some other op on this machine switches implementation by shape (not investigated here).")
else:
    print("Running alone and in a batch are identical (no implementation switch by batch); see section C for the floor.")
assert max(d.values()) < 1e-3
9.5e-07 remains after turning off the conv fast path; the linear alone differs by 9.5e-07 between batch 1 and 16.

In the outputs stored above, a small difference remains after the conv fast path is turned off, and it comes from the linear: a matrix-multiply library may take different routines for an input with “one row” vs. “many rows” (an inference; we did not read the source to confirm).

Summary

  • fp32 itself has a floor: swapping the implementation of the same conv, or running the whole model alone vs. in a batch, both differ on the order of 1e-5 to 1e-4 (the numbers printed above).
  • When comparing any two implementations (Q/DQ vs. reference, eager vs. lowered, PyTorch vs. ONNX Runtime), a difference of this magnitude is the floor of the fp32 kernels, not a fault of quantization or lowering.
  • In a quantized graph, this small difference occasionally pushes a value sitting on a rounding boundary into the neighboring step, becoming “off by exactly 1 step”, which is the phenomenon seen in M06-2.