Skip to main content

End-to-End PT2E: Quantization, Backend Deployment, and Who Does What on Edge Devices

A few words before we start 2026/11/14

These days it feels like teaching material is mostly read by AI, and AI probably already knows all of this
If AI helps with quantization and deployment, do humans — the bottleneck — still need to understand it? I don’t know, but right now I want to write down what I’ve learned from experience
Anyway, this course is written for humans, assuming anyone still wants to understand …

Use PyTorch 2 Export (PT2E) to quantize a model to int8 and other low-bit integers: start from the graph you get out of torch.export, see how the quantizer annotates it, how observers are inserted, what the graph looks like after convert, and finally hand it to a backend such as XNNPACK or ONNX Runtime.

This course targets edge devices with limited compute and power (for example always-on sensing and voice wake-up): small models, a fully integer graph (mostly int8), fixed shapes, and a limited set of ops supported by the backend. Weight-only int4 and KV-cache quantization, as commonly used for LLMs, are out of scope.

Each module starts with an overview that states the questions the chapter answers and its core concepts; the notebooks that follow verify them hands-on, one by one. All notebook outputs have already been run, so you can read the results directly; to change parameters and rerun yourself, see “Run on your own machine” below.

The storyline: two parts

M00 is the common foundation: capture the model as a graph and learn to edit it. After that the course splits into two parts, divided by the Q/DQ graph produced by convert_pt2e:

flowchart TB
    F["fp32 model"] --> P1["Part 1: M01–M06<br/>export → prepare<br/>→ calibrate or train → convert"]
    P1 --> QDQ["Q/DQ graph<br/>fp32 ops simulating int8"]
    QDQ --> P2["Part 2: M07–M10<br/>recognize dq → op → q<br/>swap in int8 kernels"]
    P2 --> D["Runs on device<br/>.pte / .onnx / custom hardware"]
    P2 -. "Quantizer, op list, constraints" .-> P1

Part 1 Part 2
Chapters M01–M06 M07–M10
Starting point A trained fp32 model The Q/DQ graph
End point The Q/DQ graph, and its accuracy An int8 model running on the device
Questions answered Where to quantize, what the scales are, what to do with QAT when PTQ is not enough Who swaps the units for int8 kernels, what to do when a swap is impossible, how to verify numerics and speed
Overview Part 1 overview Part 2 overview

The reading order is Part 1 → Part 2, but the dashed line in the diagram points the other way: the Quantizer is designed around the backend’s own kernels (M03), so in practice you pick the backend first, then quantize (the M07 overview, “Before Quantizing, Choose the Backend”). M03 reads and writes a Quantizer from the user’s point of view; M10 takes the backend’s point of view and writes hardware constraints into a Quantizer. The Quantizer is the contract between the two parts.

Learning path

Module Topic
M00 torch.export and editing the fx graph
Part 1 overview From an fp32 model to the Q/DQ graph: the storyline and the algorithm engineer’s duties
M01 Your first quantization: four steps, the Q/DQ graph
M02 Observers and fake quant: inserting observers and doing convert by hand
M03 Annotations and the Quantizer: who decides where observers go
M04 Conv + BN folding: PTQ folds at prepare, QAT folds only at convert
M05 Debugging PTQ accuracy: SQNR, calibration, per-layer diagnosis, mixed precision
M06 The QAT training flow: STE, the switches before evaluation, training graph = convert graph, QAT vs PTQ at int4
Part 2 overview After the Q/DQ graph: the storyline and the backend deployment engineer’s duties
M07 Backend deployment overview: backend layers, three routes, why the Quantizer must match the backend; what the Q/DQ graph becomes through the reference representation, lowering, and Inductor; the fp32 numeric floor
M08 XNNPACK / ExecuTorch: Q/DQ graph → lowering → .pte → runtime; unsupported ops split the graph; the gain divides into switching runtime and quantization
M09 ONNX Runtime: exporting the Q/DQ graph to QDQ ONNX; five graph-optimization levels swap in int8 kernels when the session is created; the QDQ node fusion op list; ORT’s own quantization tools
M10 A custom backend: writing the hardware constraints of a fictional VPU as a Quantizer; a check pass that only reads the Q/DQ graph; a partitioned graph that sends unsupported ops back to the CPU

The shortest reading path

If you are short on time, read these four first to get the whole picture, then dive into the chapters you need:

  1. This page: End-to-End PT2E: Quantization, Backend Deployment, and Who Does What on Edge Devices
  2. M00 overview | Turning a Model into a Graph: torch.export, ATen, and fx
  3. Part 1 overview | From an fp32 Model to the Q/DQ Graph (M01–M06)
  4. Part 2 overview | After the Q/DQ Graph: Handing Off to the Backend (M07–M10)

Two roles: the algorithm engineer and the backend deployment engineer

In practice the two parts are often owned by two kinds of people. One person may play both roles, but what gets handed over is the same.

Algorithm engineer Backend deployment engineer
Responsible for Modeling, training, quantization Choosing the platform, providing the Quantizer, lowering, verification, and deployment
Main chapters Part 1 (M01–M06) Part 2 (M07–M10)
Hands to the other side Q/DQ graph, fp32 model, data, accuracy report Quantizer, op list, hardware constraints
Owns which numbers Q/DQ graph vs fp32 Device output vs Q/DQ graph

The order of collaboration

sequenceDiagram
    participant A as Algorithm engineer
    participant B as Backend deployment engineer
    B->>A: Quantizer, op list, shape and bit-width constraints
    A->>A: Design and train the fp32 model, avoiding ops outside the list
    A->>A: PTQ or QAT, confirm accuracy on the Q/DQ graph
    A->>B: Q/DQ graph, fp32 model, data, accuracy report
    B->>B: Compliance check, lowering, fallback report
    B->>B: Numerics and speed verification
    B-->>A: Report ops that fell back, accuracy gap, speed
    A->>B: Revised Q/DQ graph

The last two steps usually go back and forth several times. The details of each handoff checklist are in the “Role” section of each part overview.

Four things this division of labor must add

Easy to miss Why Where
Give the backend information before the model is designed The op list determines how many segments the graph is split into; you can avoid that when choosing the architecture M10-3
Handing over only the Q/DQ graph is not enough The backend needs the fp32 model as a baseline and data to rerun evaluation; some routes also need to convert again M07 overview, M08-2
QAT consistency is a joint responsibility The QAT config from the backend must pass the “training graph = convert graph” check M06-3
When accuracy drops, measure by segment before deciding who investigates The loss splits into two segments, each owned by one role (see the next section) M05, M07-1

Accuracy dropped: which segment to check first

Between fp32 and the device sits the Q/DQ graph. Running three versions on the same evaluation data splits the accuracy loss into two segments:

fp32 model
  → Q/DQ graph        Segment 1: loss caused by quantization itself (algorithm engineer)
  → model on device   Segment 2: differences from lowering and integer kernels (backend deployment engineer)

Segment 1 can be measured inside PyTorch (M05, M06); segment 2 can only be measured once the backend runs the graph on the device. Here are three hypothetical accuracy sets showing how to decide:

fp32 Q/DQ graph Device Verdict
90% 82% 82% Segment 1 already lost it: the device is just faithfully reproducing the Q/DQ graph; go back and change the quantization config or the model
90% 89.5% 82% Only segment 2 lost it: the Q/DQ graph is fine; investigate lowering or the backend kernels
90% 89.5% 89.4% Both segments are fine: the small gap between device and Q/DQ graph is integer rounding

In the first case, no amount of backend tuning will recover it; in the second, more QAT by the algorithm engineer won’t help. Split by segment first, so neither side wastes effort. The “small gap” in the last row means the device and Q/DQ graph outputs differ by only 1–2 output LSBs (measured in M08 and M09).

Run on your own machine

The environment is managed with uv (Python 3.12). graphviz subgraphs need the system dot executable.

brew install graphviz          # Ubuntu: sudo apt install graphviz
uv sync
uv run jupyter lab             # or open the .ipynb in VS Code and pick .venv as the kernel