# Post-training quantization (PTQ) Quantization is an effective model optimization technique that compresses your models. Quantization with Model Optimizer can compress model size by 2x-4x, speeding up inference while preserving model quality. Model Optimizer enables highly performant quantization formats including NVFP4, FP8, INT8, INT4 and supports advanced algorithms such as SmoothQuant, AWQ, SVDQuant, and Double Quantization with easy-to-use Python APIs. This section focuses on Post-training quantization, a technique that reduces model precision after training to improve inference efficiency without requiring retraining.
| **Section** | **Description** | **Link** | **Docs** | | :------------: | :------------: | :------------: | :------------: | | Pre-Requisites | Required & optional packages to use this technique | \[[Link](#pre-requisites)\] | | | Getting Started | Learn how to optimize your models using PTQ to reduce precision and improve inference efficiency | \[[Link](#getting-started)\] | \[[docs](https://nvidia.github.io/Model-Optimizer/guides/1_quantization.html)\] | | Support Matrix | View the support matrix to see quantization compatibility and feature availability across different models | \[[Link](#support-matrix)\] | | | AutoQuantize | Automatically chooses layers/precisions for mixed precision quantization to enhanced inference performance and accuracy tradeoffs | \[[Link](#autoquantize)\] | \[[docs](https://nvidia.github.io/Model-Optimizer/guides/_pytorch_quantization.html#optimal-partial-quantization-using-auto-quantize)\] | | Real Quant | Real Quant compresses model weights in a low-precision format to reduce memory requirements of quantization. | \[[Link](https://nvidia.github.io/Model-Optimizer/guides/_compress_quantized_models.html)\] | | | Framework Scripts | Example scripts demonstrating quantization techniques for optimizing Hugging Face / Megatron-Bridge / Megatron-LM models | \[[Link](#framework-scripts)\] | | | Evaluate Accuracy | Evaluate your model's accuracy! | \[[Link](#evaluate-accuracy)\] | | | Exporting Checkpoints | Export to Hugging Face Unified Checkpoint and deploy on TRT-LLM/vLLM/SGLang | \[[Link](#exporting-checkpoints)\] | \[[docs](https://nvidia.github.io/Model-Optimizer/deployment/3_unified_hf.html)\] | | Pre-Quantized Checkpoints | Ready to deploy Hugging Face pre-quantized checkpoints | \[[Link](#pre-quantized-checkpoints)\] | | | Tracking runs with MLflow | Record a PTQ run on an MLflow server so it can be reproduced from its entry alone | \[[Link](#tracking-runs-with-mlflow)\] | | | Resources | Extra links to relevant resources | \[[Link](#resources)\] | |
## Pre-Requisites ### Docker For Hugging Face models, please use the TensorRT-LLM docker image (e.g., `nvcr.io/nvidia/tensorrt-llm/release:1.2.0`). Visit our [installation docs](https://nvidia.github.io/Model-Optimizer/getting_started/2_installation.html) for more information. Also follow the installation steps below to upgrade to the latest version of Model Optimizer and install example-specific dependencies. ### Local Installation For Hugging Face models, install Model Optimizer with `hf` dependencies using `pip` from [PyPI](https://pypi.org/project/nvidia-modelopt/) and install the requirements for the example: ```bash pip install -U nvidia-modelopt[hf] pip install -r requirements.txt ``` For TensorRT-LLM deployment, please use the TensorRT-LLM docker image or follow their [installation docs](https://nvidia.github.io/TensorRT-LLM/installation/index.html). Similarly, for vLLM or SGLang deployment, please use their installation docs. ## Getting Started ### 1. Quantize (Post Training Quantization) With the simple API below, you can very easily use Model Optimizer to quantize your model. Model Optimizer achieves this by converting the precision of your model to the desired precision, and then using a small dataset (typically 128-512 samples) to [calibrate](https://nvidia.github.io/Model-Optimizer/guides/_basic_quantization.html) the quantization scaling factors. The accuracy of PTQ is typically robust across different choices of calibration data, by default Model Optimizer uses a mix of [`cnn_dailymail`](https://huggingface.co/datasets/abisee/cnn_dailymail) and [`nemotron-post-training-dataset-v2`](https://huggingface.co/datasets/nvidia/Nemotron-Post-Training-Dataset-v2). Users can try other datasets by easily modifying the `calib_set`. ```python import modelopt.torch.quantization as mtq # Setup the model model = AutoModelForCausalLM.from_pretrained("...") # Simplified example set up a calibration data loader with the desired calib_size calib_set = get_dataloader(num_samples=calib_size) # Prepare the calibration set and define a forward loop def forward_loop(model): for batch in calib_set: model(batch) # PTQ with in-place replacement to quantized modules model = mtq.quantize(model, mtq.NVFP4_DEFAULT_CFG, forward_loop) ``` > *For higher NVFP4 PTQ accuracy, we recommend using `mtq.NVFP4_MLP_ONLY_CFG`, `mtq.NVFP4_EXPERTS_ONLY_CFG`, or `mtq.NVFP4_OMLP_ONLY_CFG` instead of `mtq.NVFP4_DEFAULT_CFG`. `NVFP4_MLP_ONLY_CFG` applies NVFP4 quantization to MLP (and MoE) layers, leaving attention layers unquantized. `NVFP4_EXPERTS_ONLY_CFG` quantizes only expert layers (`*mlp.experts*` and `*block_sparse_moe*`), useful for MoE models where dense MLP and attention stay in higher precision. `NVFP4_OMLP_ONLY_CFG` additionally quantizes the `o_proj` layer. All preserve accuracy in the sensitive attention QKV projections while still providing significant compression.* ### 2. Export Quantized Model Once your model is quantized, you can now export that model to a checkpoint for easy deployment. \ We provide two APIs to export the quantized model: - Unified Hugging Face checkpoints, which can be deployed on TensorRT-LLM (Pytorch and C++ backends), [vLLM](https://github.com/vllm-project/vllm) and [SGLang](https://github.com/sgl-project/sglang). - (Legacy) TensorRT-LLM checkpoints, a format that works with TensorRT-LLM C++ backend only. #### Unified Hugging Face Checkpoints ```python from modelopt.torch.export import export_hf_checkpoint with torch.inference_mode(): export_hf_checkpoint( model, # The quantized model. export_dir, # The directory where the exported files will be stored. ) ``` Please reference our [framework scripts](#framework-scripts) and our [docs](https://nvidia.github.io/Model-Optimizer/guides/1_quantization.html) for more details. ## Support Matrix ### Hugging Face Supported Models | Model | fp8 | int8_smoothquant | int4_awq | w4a8_awq_beta1 | nvfp45 | | :---: | :---: | :---: | :---: | :---: | :---: | | LLAMA 3.x | ✅ | ❌ | ✅ | ✅3 | ✅ | | LLAMA 4 6 | ✅ | ❌ | ❌ | ❌ | ✅ | | Mixtral | ✅ | ❌ | ✅2 | ❌ | ✅ | | Phi-3,4 | ✅ | ✅ | ✅ | ✅3 | - | | Phi-3.5 MOE | ✅ | ❌ | ❌ | ❌ | - | | Llama-Nemotron Super | ✅ | ❌ | ❌ | ❌ | ✅ | | Llama-Nemotron Ultra | ✅ | ❌ | ❌ | ❌ | ❌ | | Gemma 3 | ✅2 | - | ✅ | - | - | | QWen 2, 2.5 4 | ✅ | ✅ | ✅ | ✅ | ✅ | | QWen3, 3.5 MOE, Next 6 | ✅ | - | - | - | ✅ | | QwQ | ✅ | - | - | - | ✅ | | DeepSeek V3, R1, V3.1, V3.27 | - | - | - | - | ✅ | | Kimi K315 | - | - | - | - | ✅ | | GLM-4.78 | ✅ | - | - | - | ✅ | | Kimi K2 | - | - | - | - | ✅ | | MiniMax M2.1 | - | - | - | - | ✅ | | GPT-OSS10 | - | - | - | - | ✅ | | T5 | ✅ | ✅ | ✅ | ✅ | - | | Whisper9 | ✅ | ❌ | ❌ | ❌ | - | | Nemotron-3 | ✅ | ❌ | ❌ | ❌ | ✅ | | Llava (VLM)11 | ✅ | ✅12 | ✅ | ✅ | - | | Qwen2, 2.5-VL (VLM)11 | ✅ | ✅12 | ✅ | ✅ | ✅ | | Qwen3-VL, Qwen3.5 (VLM)11,14 | ✅ | - | - | - | - | | Gemma 3 (VLM)11 | ✅ | - | - | - | - | | Nemotron VL (VLM)11,13 | ✅ | - | - | - | ✅ | > *This is a subset of the models supported. For the full list please check the [TensorRT-LLM support matrix](https://nvidia.github.io/TensorRT-LLM/reference/precision.html#support-matrix)* > *1.The w4a8_awq_beta is an experimental quantization scheme that may result in a higher accuracy penalty.* \ > *2.For some models, there is only support for exporting quantized checkpoints.* \ > *3.W4A8_AWQ is only available on some models but not all* \ > *4.For some models, KV cache quantization may result in a higher accuracy penalty.* \ > *5.A selective set of the popular models are internally tested. The actual model support list may be longer. NVFP4 inference requires Blackwell GPUs and TensorRT-LLM v1.2 or later* \ > *6.Some models currently support export to HF format only.* \ > *7.[PTQ for DeepSeek](../deepseek/README.md)* \ > *8.GLM-4.7 has MTP (Multi-Token Prediction) layers that are automatically loaded and excluded from quantization.* \ > *9.Running Whisper model with transformers>=5.0 requires [torchcodec](https://github.com/meta-pytorch/torchcodec?tab=readme-ov-file#installing-cuda-enabled-torchcodec) and other system packages (e.g. ffmpeg).* \ > *10.GPT-OSS ships with native MXFP4 weights; NVFP4 export is produced via the closed-form `--cast_mxfp4_to_nvfp4` cast (see [MXFP4 → NVFP4 cast](#mxfp4--nvfp4-cast-for-gpt-oss)).* \ > *11.Vision-language model (VLM): by default, only the language model is quantized while the vision encoder is kept in high precision. Pass `--vlm` to the shell script (see [VLM quantization](#vlm-quantization)).* \ > *12.For VLMs, `int8_smoothquant` only supports TensorRT-LLM checkpoint export and is not compatible with the TensorRT-LLM torch backend.* \ > *13.Nemotron VL automatically calibrates with image-text pairs; see [VLM calibration with image-text pairs](#vlm-calibration-with-image-text-pairs-eg-nemotron-vl).* \ > *14.Qwen3-VL and dense Qwen3.5 VLM checkpoints support opt-in FP8 vision encoder quantization through model-specific recipes. Vision Linear layers, including primary and deepstack merger Linears where present, are quantized; patch embedding and vision-attention BMMs remain in high precision. MoE variants are not validated by these recipes.* \ > *15.Kimi K3 uses the calibration-free [streaming converter](../kimi/README.md) because its routed experts are released as packed MXFP4 tensors; it does not use the in-memory `hf_ptq.py` flow.* > *The accuracy loss after PTQ may vary depending on the actual model and the quantization method. Different models may have different accuracy loss and usually the accuracy loss is more significant when the base model is small. If the accuracy after PTQ is not meeting the requirement, please try either modifying [hf_ptq.py](./hf_ptq.py) and disabling the KV cache quantization or using the [QAT](./../llm_qat/README.md) instead. For NVFP4 quantization specifically, we recommend `nvfp4_mlp_only`, `nvfp4_experts_only`, or `nvfp4_omlp_only` to achieve higher accuracy by restricting quantization to the MLP/expert layers (and optionally the `o_proj` layer) while keeping the attention QKV projections unquantized.* > You can also create your own custom config using [this](https://nvidia.github.io/Model-Optimizer/guides/_pytorch_quantization.html#custom-calibration-algorithm) guide. > *Vision-language models (VLMs) are listed in the support matrix above (rows marked `(VLM)`). PTQ for > VLMs is handled by the same `hf_ptq.py` entry point and shell script as LLMs. By default, the > language model is quantized while the vision branch remains in high precision. Dense Qwen3-VL > and dense Qwen3.5 VLM checkpoints additionally support the opt-in FP8 recipes documented under > [VLM quantization](#vlm-quantization). For detailed TensorRT-LLM torch backend multimodal support, > please refer to [this doc](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/models/supported-models.md#multimodal-feature-support-matrix-pytorch-backend).* ## Framework Scripts ### Hugging Face Example [Script](./scripts/huggingface_example.sh) For LLM models like [Llama-3](https://huggingface.co/meta-llama): ```bash # Install model specific pip dependencies if needed export HF_PATH= scripts/huggingface_example.sh --model $HF_PATH --quant --tp [1|2|4|8] ``` `QFORMAT` accepts any preset basename under [`modelopt_recipes/configs/ptq/presets/model/`](../../modelopt_recipes/configs/ptq/presets/model) — e.g. `fp8`, `fp8_per_channel_per_token`, `fp8_2d_blockwise_weight_only`, `int8`, `int8_smoothquant`, `int8_weight_only`, `int4_awq`, `w4a8_awq_beta`, `nvfp4`, `nvfp4_awq_lite`, `nvfp4_w4a4_weight_mse_fp8_sweep`, `nvfp4_mlp_only`, `nvfp4_experts_only`, `nvfp4_omlp_only`, `nvfp4_svdquant`, `nvfp4_w4a4_weight_local_hessian`, `w4a8_nvfp4_fp8`, `w4a8_mxfp4_fp8`, `mxfp8`. > *By default `trust_remote_code` is set to false. Please turn it on if model calibration and eval requires it using `--trust_remote_code`.* > *If the Huggingface model calibration fails on a multi-GPU system due to mismatched tensor placement, please try setting CUDA_VISIBLE_DEVICES to a smaller number.* > *FP8 calibration over a large model with limited GPU memory is not recommended but possible with the [accelerate](https://huggingface.co/docs/accelerate/en/usage_guides/big_modeling) package. Please tune the device_map setting in [`example_utils.py`](./example_utils.py) if needed for model loading and the calibration process can be slow.* > *Huggingface models trained with `modelopt.torch.speculative` can be used as regular Huggingface models in PTQ. Note: there is a known issue with Huggingface models loaded across multiple GPUs for inference (i.e., "Expected all tensors to be on the same device, but found at least two devices..."). When encountered this error in PTQ of speculative decoding models, try reducing the number of GPUs used.* > *Calibration by default uses left padding_side for the Huggingface tokenizer as it usually leads to lower accuracy loss. The exported tokenizer files restores the default padding_side.* > *If a GPU OOM error occurs during model quantization despite sufficient memory, setting the --use_seq_device_map flag can help. This enforces sequential device mapping, distributing the model across GPUs and utilizing up to 80% of each GPU's memory.* > *You can add `--low_memory_mode` to the command to lower the memory requirements of the PTQ process. With this mode, the script will compress model weights to low precision before calibration. This mode is only supported for FP8 and NVFP4 with max calibration.* #### Recipe-based Quantization Instead of specifying `--qformat` and `--kv_cache_qformat` separately, you can use a **recipe** — a declarative YAML file that bundles the full quantization configuration. Recipes are loaded via `--recipe` and take precedence over `--qformat`. ```bash # Using a built-in recipe name (without .yaml suffix) python hf_ptq.py \ --pyt_ckpt_path \ --recipe general/ptq/nvfp4_default-kv_fp8_cast \ --export_path # Using a custom recipe YAML file path python hf_ptq.py \ --pyt_ckpt_path \ --recipe /path/to/my_ptq.yaml \ --export_path ``` Built-in recipes are located in `modelopt_recipes/general/ptq/` for model-agnostic recipes and in `modelopt_recipes/model_type//ptq/` for recipes tuned to a specific Hugging Face `model_type` (see [`modelopt_recipes/model_type/README.md`](../../modelopt_recipes/model_type/README.md)). You can also provide a path to your own custom YAML recipe file or directory. See the [recipe documentation](https://nvidia.github.io/Model-Optimizer) for details on the YAML schema and available recipes. > *When `--recipe` is specified, `--qformat` is ignored. KV cache handling depends on the recipe type: a **PTQ** recipe bakes KV cache into its config and ignores `--kv_cache_qformat`; an **AutoQuantize** recipe falls back to `--kv_cache_qformat` unless it sets an explicit `kv_cache` field.* #### KV Cache Quantization KV cache quantization reduces memory usage during inference by quantizing the key-value cache. This is controlled via the `--kv_cache_qformat` flag (default: `fp8_cast`). ```bash # FP8 KV cache with cast (no calibration needed, fast) python hf_ptq.py --pyt_ckpt_path --qformat fp8 --kv_cache_qformat fp8_cast --export_path # NVFP4 KV cache with data-driven calibration python hf_ptq.py --pyt_ckpt_path --qformat nvfp4 --kv_cache_qformat nvfp4 --export_path # Disable KV cache quantization python hf_ptq.py --pyt_ckpt_path --qformat fp8 --kv_cache_qformat none --export_path ``` Via the shell script: ```bash scripts/huggingface_example.sh --model $HF_PATH --quant fp8 --kv_cache_quant nvfp4 ``` Available KV cache formats: | Format | Description | | :---: | :--- | | `fp8_cast` (default) | FP8 KV cache without data-driven calibration (amax set to FP8 range) | | `fp8` | FP8 KV cache with data-driven calibration | | `fp8_affine` | FP8 KV cache with affine quantization | | `nvfp4_cast` | NVFP4 KV cache without data-driven calibration | | `nvfp4` | NVFP4 KV cache with data-driven calibration | | `nvfp4_affine` | NVFP4 KV cache with affine quantization | | `nvfp4_rotate` | NVFP4 KV cache with rotation | | `none` | Disable KV cache quantization | > *Formats ending in `_cast` (fp8_cast, nvfp4_cast) are fast — they set the amax to the format's full range without data-driven calibration. Other formats use data-driven calibration for potentially better accuracy.* #### MXFP4 → NVFP4 cast (for GPT-OSS) GPT-OSS checkpoints (`openai/gpt-oss-20b`, `openai/gpt-oss-120b`) ship with native MXFP4 weights (`*_blocks` + `*_scales` in the checkpoint, `quantization_config.quant_method == "mxfp4"`). Passing `--cast_mxfp4_to_nvfp4` tells `hf_ptq.py` to read the source MXFP4 scales and produce a closed-form, bit-exact NVFP4 weight export — no GEMM-level recalibration of the weights needed. ```bash python hf_ptq.py \ --pyt_ckpt_path openai/gpt-oss-20b \ --qformat nvfp4_mlp_only \ --cast_mxfp4_to_nvfp4 \ --export_path ``` The cast pins each NVFP4 block's `scale_2 = 2^(k_max - 8)` and `_amax = 6 * 2^k_j`, both derived from the source MXFP4 E8M0 scales. For blocks whose `k_j` lands in E4M3's representable window (`k_max - k_j ≤ 17`), NVFP4 dequant matches MXFP4 dequant bit-for-bit; out-of-range blocks fall back to a data-derived per-block amax. > *`--cast_mxfp4_to_nvfp4` requires an NVFP4-family `--qformat` (e.g. `nvfp4_mlp_only`, `nvfp4_experts_only`, `nvfp4`) and is incompatible with AutoQuantize recipes (multi-format search).* #### Deepseek R1 [PTQ for DeepSeek](../deepseek/README.md) shows how to quantize the DeepSeek model with FP4 and export to TensorRT-LLM. #### Kimi K3 [PTQ for Kimi K3](../kimi/README.md) shows how to cast the source MXFP4 routed experts to NVFP4 and quantize KDA/MLA attention weights to 128x128 block FP8 without loading the full model or running calibration. #### VLM quantization Vision-language models are quantized through the same script. Add `--vlm` so the script runs the TensorRT-LLM multimodal quickstart as the deploy smoke test instead of the text-only one: ```bash scripts/huggingface_example.sh --model --quant fp8 --vlm ``` Supported `--quant` values for VLMs are `fp8`, `nvfp4`, `int8_smoothquant`, `int4_awq`, and `w4a8_awq_beta` (see the `(VLM)` rows in the [Support Matrix](#hugging-face-supported-models)). By default, `hf_ptq.py` applies `--qformat` only to the language model. Model-specific recipes add FP8 quantization of the vision branch for validated Qwen3-VL and dense Qwen3.5 checkpoints. Use the recipe directory matching the checkpoint's `model_type`: `qwen3_vl` or `qwen3_5`. ```bash # Vision encoder only: FP8 vision Linears and merger, BF16 LLM and KV cache. python hf_ptq.py \ --pyt_ckpt_path \ --recipe model_type/qwen3_vl/ptq/fp8_vision-kv_none \ --calib_with_images \ --calib_size 512 \ --skip_generate \ --export_path # Joint vision encoder + language model FP8 with FP8 KV-cache cast. python hf_ptq.py \ --pyt_ckpt_path \ --recipe model_type/qwen3_vl/ptq/fp8_vision_lm-kv_fp8_cast \ --calib_with_images \ --calib_size 512 \ --skip_generate \ --export_path ``` `fp8_vision-kv_none` starts from all quantizers disabled and enables FP8 only for `nn.Linear` weights and inputs under the `visual` branch, including the primary merger and any deepstack mergers. The language model, KV cache, patch embedding, and vision-attention QK/softmax/AV operations stay in high precision. `fp8_vision_lm-kv_fp8_cast` applies the standard FP8 model recipe to both model branches and enables FP8 KV-cache cast, while keeping patch embedding and vision-attention operands in high precision. Other precision combinations can be expressed by composing the same recipe units in a custom recipe; no model-specific Python path is required. The exported checkpoint requires an inference runtime that supports quantized vision encoder Linears. Runtime-specific vision-attention quantization is separate from this ModelOpt checkpoint. Use the direct `hf_ptq.py` commands above for these recipes; the generic multimodal quickstart is not a serving validation for a vision-quantized checkpoint. Both examples pass `--skip_generate` because the script's text-only preview does not forward the image tensors used during calibration. For a Qwen3.5 checkpoint, replace `qwen3_vl` with `qwen3_5` in the recipe path. #### VLM calibration with image-text pairs (e.g., Nemotron VL) For vision-language models, calibration quality can likely improve by using image-text pairs instead of text-only data, especially on visual understanding tasks: ```bash python hf_ptq.py \ --pyt_ckpt_path \ --qformat nvfp4 \ --export_path \ --trust_remote_code \ --calib_with_images \ --calib_size 512 ``` The same flag is exposed by the shell script: ```bash scripts/huggingface_example.sh --model --quant nvfp4 --vlm --calib_with_images --trust_remote_code ``` With `--calib_with_images`, calibration batches always pass through the complete VLM so image features reach the component selected by the preset or recipe. This also applies to the default language-model-only path: its quantizers are exercised by the complete multimodal forward. > Note: when `--calib_with_images` is set, `--calib_size` must be a single value, and the calibration dataset is nvidia/nemotron_vlm_dataset_v2. This functionality is currently in beta and has been tested on `nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16`. ### Megatron-Bridge Example Script Please refer to [examples/megatron_bridge/README.md](../megatron_bridge/README.md) for example scripts for PTQ / QAD with Megatron-Bridge which is generally more performant than the Hugging Face scripts. ### Megatron-LM Example Script Megatron-LM framework PTQ and TensorRT-LLM deployment examples are maintained in the Megatron-LM GitHub repo. Please refer to the examples [here](https://github.com/NVIDIA/Megatron-LM/tree/main/examples/post_training/modelopt). ## AutoQuantize [AutoQuantize (`mtq.auto_quantize`)](https://nvidia.github.io/Model-Optimizer/reference/generated/modelopt.torch.quantization.model_quant.html#modelopt.torch.quantization.model_quant.auto_quantize) is a PTQ algorithm which quantizes a model by searching for the best quantization format per-layer while meeting performance constraints specified by the user. `AutoQuantize` streamlines the trade-off of model accuracy and performance. `AutoQuantize` uses an effective-bits target (`effective_bits`) as the performance constraint (for both weight-only and weight & activation quantization) — the effective number of bits for the quantized model. You may specify an `effective_bits` target such as 5.4 for mixed precision quantization using `NVFP4_DEFAULT_CFG` & `FP8_DEFAULT_CFG`. `AutoQuantize` will automatically quantize highly sensitive layers in `FP8_DEFAULT_CFG` while keeping less sensitive layers in `NVFP4_DEFAULT_CFG` (and even skip quantization for any extremely sensitive layers) so that the the final mixed precision quantized model has an effective quantized bits of 5.4. This model would give a better accuracy than the model quantized with vanilla `NVFP4_DEFAULT_CFG` configuration since the more aggressive `NVFP4_DEFAULT_CFG` quantization was not applied for the highly sensitive layers. Here is an example usage for `AutoQuantize` algorithm (Please see [auto_quantize](https://nvidia.github.io/Model-Optimizer/reference/generated/modelopt.torch.quantization.model_quant.html#modelopt.torch.quantization.model_quant.auto_quantize) API for more details): ```python import modelopt.torch.quantization as mtq # Define the model & calibration dataloader model = ... calib_dataloader = ... # Define forward_step function. # forward_step should take the model and data as input and return the output def forward_step(model, data): output = model(data) return output # Define loss function which takes the model output and data as input and returns the loss def loss_func(output, data): loss = ... return loss # Perform AutoQuantize model, search_state_dict = mtq.auto_quantize( model, constraints = {"effective_bits": 5.4}, # supported quantization formats are listed in `modelopt.torch.quantization.config.choices` quantization_formats = ["NVFP4_DEFAULT_CFG", "FP8_DEFAULT_CFG"] data_loader = calib_dataloader, forward_step=forward_step, loss_func=loss_func, ... ) ``` ### AutoQuantize for Hugging Face models `AutoQuantize` can be performed for Huggingface LLM models like [Qwen](https://huggingface.co/Qwen/Qwen3-8B) / [Nemotron](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16) as shown below: `AutoQuantize` is driven by an **AutoQuantize recipe** passed with `--recipe`. The recipe defines the candidate formats, optional fixed PTQ baseline, `effective_bits` target, cost model, scoring method, search-disabled layers, and cost-excluded layers — see [`AutoQuantizeConfig`](../../modelopt/recipe/config.py). Shipped recipes live in [`modelopt_recipes/general/auto_quantize/`](../../modelopt_recipes/general/auto_quantize); model-specific recipes (carrying architecture-specific disabled layers — e.g. VL vision towers) live under `modelopt_recipes/model_type//auto_quantize/`. [Script](./scripts/huggingface_example.sh) ```bash export HF_PATH= # --recipe selects an AutoQuantize recipe; the recipe defines the candidate formats and the # effective-bits target (here NVFP4 + FP8 at 5.4 effective bits). scripts/huggingface_example.sh --model $HF_PATH --recipe general/auto_quantize/nvfp4_fp8_at_5p4bits --calib_batch_size 4 ``` The recipe quantizes the less accuracy-sensitive layers with the more aggressive format (e.g. NVFP4) and keeps the more sensitive ones at higher precision (or unquantized), so the model meets the recipe's `effective_bits` target. To author your own, copy a shipped recipe and adjust `candidate_formats`, `constraints.effective_bits`, `auto_quantize_method` (`gradient` / `kl_div`), `score_size`, `module_search_spaces` (optional per-module candidate overrides), `disabled_layers` (excluded from the search), and `cost_excluded_layers` (kept out of the bit-budget accounting — e.g. VL vision towers). Recipes can splice a shared base `disabled_layers` set via `$import` (see `modelopt_recipes/configs/auto_quantize/units/base_disabled_layers`). AutoQuantize recipes support two mutually exclusive search-space styles: 1. Set top-level `auto_quantize.candidate_formats` to search every unmatched quantizable module, with optional `module_search_spaces` overrides. 2. Set a normal top-level `quantize` config as the fixed PTQ baseline, omit top-level `candidate_formats`, and use `module_search_spaces` to list only the modules AutoQuantize should search. The fixed and searched modules still run through one integrated calibration, scoring, cost, and export flow. For example, this keeps unmatched modules at the normal W4A16 NVFP4 PTQ setting while searching only attention between W4A16 and FP8: ```yaml imports: w4a16_nvfp4: configs/ptq/presets/model/w4a16_nvfp4 fp8: configs/ptq/presets/model/fp8 quantize: $import: w4a16_nvfp4 auto_quantize: constraints: effective_bits: 6.0 module_search_spaces: - module_name_patterns: ["*self_attn*", "*linear_attn*"] candidate_formats: - $import: w4a16_nvfp4 - $import: fp8 allow_no_quant: false ``` bf16 (no quantization) is an implicit per-layer choice for the top-level `candidate_formats`, so a single format (e.g. `[fp8]`) gives a `{fp8, bf16}` per-layer search. A `module_search_spaces` rule can set `allow_no_quant: false` to exclude bf16 from the solver choices for matching modules. Use the top-level `quantize` baseline, rather than a one-candidate search rule, for modules that are fixed and not actually searched. The fixed baseline may also reuse a model-specific PTQ configuration. For example, the Qwen3.6 MoE AutoQuantize recipe imports the same model-specific `quant_cfg` used by `model_type/qwen3_5_moe/ptq/w4a16_nvfp4-fp8_attn-kv_fp8_cast`, reproduces that recipe's `quantize` section, and lists only shared experts, attention, and `lm_head` under `module_search_spaces`. A loader test asserts that the inherited fixed baseline remains equal to the original PTQ recipe while leaving the original recipe unchanged. For models without backprop support (e.g. Llama-4), use the `kl_div` scoring method — see the shipped `general/auto_quantize/nvfp4_fp8_kl_div_at_5p4bits` recipe. Weight AutoQuantize recipes still apply KV cache as a uniform post-step and fall back to `--kv_cache_qformat` (default `fp8_cast`) unless they set an explicit `kv_cache` field. KV-cache AutoQuantize recipes use the same `mtq.auto_quantize` API and set `constraints.cost_model: kv_cache` with an `effective_bits` target. Their `candidate_formats` are complete K/V cache configs whose config-level `effective_bits` includes packed scale overhead. The width-weighted budget covers eligible layers; `disabled_layers` are preserved and excluded. BF16 is used only as the isolated-KL reference, not as a solver choice. The shipped recipe searches FP8-cast K/V (8.0 bits/scalar) and NVFP4-cast K/V (4.5 bits/scalar) at 5.4 bits/scalar. It intentionally excludes FP8-K/NVFP4-V because the companion vLLM implementation does not support that asymmetric per-layer format: ```bash python hf_ptq.py \ --pyt_ckpt_path Qwen/Qwen3.8-27B \ --recipe general/auto_quantize/kv_fp8_nvfp4_cast_kl_div_at_5p4bits \ --auto_quantize_checkpoint /path/to/kv_autoquant.pth \ --export_path /path/to/qwen3.8-27b-mixed-kv ``` Each candidate uses an explicit constant scale, avoiding an additional calibration pass while keeping persistent K/V scales in the unified HF checkpoint. Unified export records the selected formats in `kv_cache_quantized_layers`. `mtq.auto_quantize` returns the sensitivity scores and selected recipe in its search state; `--auto_quantize_checkpoint` stores that resumable state, including the candidate quantizer tensors needed for replay. KV sensitivity scoring runs one reference forward plus one forward per eligible-layer candidate for every scoring step. Choose the recipe's `auto_quantize.score_size` with the number of eligible layers and candidates in mind. Peak scoring memory also grows with the number of selected tokens times the model vocabulary because reference and candidate log probabilities are computed in FP32. > [!NOTE] > Layer-wise KV checkpoints require the companion > [vLLM mixed-KV metadata consumer](https://github.com/vllm-project/vllm/pull/52813) or a later > vLLM release containing it. The repository's currently pinned vLLM 0.26.0 does not consume > `kv_cache_quantized_layers`, so these checkpoints are export-only in that stock environment. > Do not deploy them with the pinned runtime. Full FP8 K/V and full NVFP4 K/V use existing vLLM > kernels once the layer-wise metadata consumer is available. The one runtime flag is `--auto_quantize_checkpoint` — save/restore the search state to resume an interrupted search (skips re-scoring): ```bash scripts/huggingface_example.sh --model $HF_PATH --recipe general/auto_quantize/nvfp4_fp8_at_5p4bits \ --auto_quantize_checkpoint /path/to/auto_quantize.pth --calib_batch_size 4 ``` The example scripts above also have an additional flag `--tasks`, where the actual tasks run in the script can be customized. The allowed tasks are `quant,mmlu,lm_eval,livecodebench,simple_eval` specified in the script [parser](./scripts/parser.sh). The tasks combo can be specified with a comma-separated task list. Some tasks like mmlu can take a long time to run. To run lm_eval tasks, please also specify the `--lm_eval_tasks` flag with comma separated lm_eval tasks [here](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks). > *If GPU out-of-memory error is reported running the scripts, please try editing the scripts and reducing the max batch size to save GPU memory.* > *NOTE: AutoQuantize requires backpropagation of the model. Models without backpropagation support (e.g., Llama-4) will not work with AutoQuantize when using the `gradient` method. The `kl_div` method does not require backpropagation.* ## Real Quant When working with large language models, memory constraints can be a significant challenge. ModelOpt provides a workflow for initializing HF models with compressed weights across multiple GPUs to dramatically reduce memory usage. Check `--low_memory_mode` option in hf_ptq.py for more details. ```python import modelopt.torch.quantization as mtq from modelopt.torch.quantization.plugins import init_quantized_weights from transformers import AutoModelForCausalLM, AutoConfig # Step 1: Initialize the model with compressed weights with init_quantized_weights(mtq.NVFP4_DEFAULT_CFG): model = AutoModelForCausalLM.from_pretrained(ckpt_path) # Step 2: Calibrate the model mtq.calibrate(model, algorithm="max", forward_loop=calibrate_loop) ``` ## Multi-Node Post-Training Quantization with FSDP2 ModelOpt enables quantization of LLMs across multiple GPU nodes using FSDP2 for distributed model sharding and calibration, exposed via the `--use_fsdp2` flag on the standard `hf_ptq.py` entry point. > *KV-cache AutoQuantize recipes are not supported with `--use_fsdp2` and are rejected before model loading. Distributed KV sensitivity scoring, selection, and checkpoint writes must be synchronized before this combination can be enabled safely. Existing weight AutoQuantize recipes retain their previous experimental warning with FSDP2.* ### Usage #### Slurm (recommended) Slurm orchestrates launching the job on every node for you, so this is the easiest way to run a multi-node PTQ. A ready-to-run example that quantizes Nemotron-3-Super to NVFP4 is provided in [`slurm/multinode_fsdp2_ptq.slurm`](./slurm/multinode_fsdp2_ptq.slurm). Edit the `CONFIG` block (container image, model path, export path, recipe) and submit: ```bash sbatch --nodes=2 slurm/multinode_fsdp2_ptq.slurm ``` #### Manual (run on each node) Without Slurm, start `torchrun` on every node yourself: ```bash torchrun \ --nnodes= --node_rank= \ --master_addr= --master_port= \ --nproc_per_node= \ hf_ptq.py \ --pyt_ckpt_path \ --recipe general/ptq/nvfp4_default-kv_fp8_cast \ --batch_size \ --calib_size \ --export_path \ --use_fsdp2 ``` See [Recipe-based Quantization](#recipe-based-quantization) for the recipe format and built-in recipe names. The exported checkpoint can be deployed using TensorRT-LLM/ vLLM/ SGLang. For more details refer to the [deployment section](#deployment) of this document. > *Performance Note: FSDP2 is designed for training workloads and may result in longer calibration and export times. For faster calibration, maximize the batch size based on available GPU memory and choose the right number of GPUs to avoid unnecessary communication.* ## Evaluate Accuracy ### TensorRT-LLM Validation A list of accuracy validation benchmarks are provided in the [llm_eval](../llm_eval/README.md) directory. Right now MMLU is supported in this example by specifying the `--tasks` flag running the scripts mentioned above. The `benchmark_suite.py` script is used as a fast performance benchmark. For details, please refer to the [TensorRT-LLM documentation](https://github.com/NVIDIA/TensorRT-LLM/blob/main/benchmarks/) This example also covers the [lm_evaluation_harness](https://github.com/EleutherAI/lm-evaluation-harness), MMLU and the human eval accuracy benchmarks, whose details can be found [here](../llm_eval/README.md). The supported lm_eval evaluation tasks are listed [here](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks) ## Exporting Checkpoints Model Optimizer supports provide two paths to export the quantized model: - Unified Hugging Face checkpoints, which can be deployed on TensorRT-LLM (Pytorch and C++ backends), [vLLM](https://github.com/vllm-project/vllm) and [SGLang](https://github.com/sgl-project/sglang). - (Legacy) TensorRT-LLM checkpoints, a format that works with TensorRT-LLM C++ backend only. The unified checkpoint1 format design reflects two key characteristics: 1. The layer structures and tensor names remain aligned with the original Hugging Face checkpoint, and 2. The same checkpoint can be deployed across multiple inference frameworks without modification. A unified checkpoint can be exported using the following commands: > *1.Unified checkpoint export currently does not support sparsity. Speculative decoding is only supported in unified checkpoint export. For legacy deployment, exported unified checkpoint then needs a TensorRT-LLM checkpoint converter (e.g., [this](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/eagle/convert_checkpoint.py)) to convert and build the TensorRT engine(s) for deployment. Alternatively, call TensorRT-LLM LLM-API to deploy the unified checkpoints e.g., check examples [here](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/llm-api/README.md).* ### API ```python from modelopt.torch.export import export_hf_checkpoint with torch.inference_mode(): export_hf_checkpoint( model, # The quantized model. export_dir, # The directory where the exported files will be stored. ) ``` ### Quantize and Export ```bash python hf_ptq.py --pyt_ckpt_path --qformat fp8 --export_path --trust_remote_code ``` > *For exporting fake-quantized models for vLLM serving (e.g., for research or kernels not yet supported in real-quant), use the `--vllm_fakequant_export` flag. See [vllm_serve/README.md](../vllm_serve/README.md) for details.* ### Hugging Face framework [Script](./scripts/huggingface_example.sh) Alternatively, the framework script `huggingface_example.sh` also supports quantize and export: ```bash scripts/huggingface_example.sh --model --quant fp8 ``` ### Deployment ______________________________________________________________________ #### TRT-LLM ```python from tensorrt_llm import LLM llm_fp8 = LLM(model="") print(llm_fp8.generate(["What's the age of the earth? "])) ``` #### vLLM ```python from vllm import LLM llm_fp8 = LLM(model="", quantization="modelopt") print(llm_fp8.generate(["What's the age of the earth? "])) ``` #### SGLang ```python import sglang as sgl llm_fp8 = sgl.Engine(model_path="", quantization="modelopt") print(llm_fp8.generate(["What's the age of the earth? "])) ``` ### Unified HF Checkpoint Deployment Model Support Matrix The deployment support matrix — which model families and quantization formats are covered on TRT-LLM, vLLM, and SGLang, including vision-language models, speculative decoding drafters, and diffusion models — lives in the documentation so there is a single copy to keep current: **[Unified HF Checkpoint → Model Support Matrix](https://nvidia.github.io/Model-Optimizer/deployment/3_unified_hf.html#model-support-matrix)** Each entry there is drawn from [`tests/examples/hf_ptq/test_deploy.py`](../../tests/examples/hf_ptq/test_deploy.py), which loads the exported checkpoint in each framework and generates from short text prompts. That file is also the place to look for the exact checkpoint, tensor-parallel size, and minimum SM version behind each entry. > *Note: those cases are marked `release` and run out-of-band — no workflow currently passes > `--run-release` — and each is a load-and-generate smoke check on the text path. Read the legend in > the docs before treating an entry as verified support.* > *Note: the matrix records what modelopt validates, not the full set of what will run. vLLM, SGLang, > and TRT-LLM load unified HF checkpoints generically, so unlisted models frequently deploy without > any modelopt change — check the serving framework's own model support list and try it.* ### (Legacy) TensorRT-LLM Checkpoints The user can specify the inference time TP and PP size and the export API will organize the weights to fit the target GPUs. ```python from modelopt.torch.export.trtllm import export_tensorrt_llm_checkpoint with torch.inference_mode(): export_tensorrt_llm_checkpoint( model, # The quantized model. decoder_type, # The type of the model, e.g gpt, gptj, or llama. dtype, # The exported weights data type. export_dir, # The directory where the exported files will be stored. inference_tensor_parallel, # The number of GPUs used in the inference time tensor parallel. inference_pipeline_parallel, # The number of GPUs used in the inference time pipeline parallel. use_nfs_workspace, # If exporting in a multi-node setup, please specify a shared directory like NFS for cross-node communication. ) ``` ### Build the TensorRT-LLM engines After the TensorRT-LLM checkpoint export, you can use the `trtllm-build` build command to build the engines from the exported checkpoints. Please check the [TensorRT-LLM Build API](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/architecture/workflow.md#build-apis) documentation for reference. ## Pre-Quantized Checkpoints - Ready-to-deploy checkpoints \[[🤗 Hugging Face - Nvidia Model Optimizer Collection](https://huggingface.co/collections/nvidia/inference-optimized-checkpoints-with-model-optimizer)\] - Deployable on [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM), [vLLM](https://github.com/vllm-project/vllm) and [SGLang](https://github.com/sgl-project/sglang) - More models coming soon! ## Tracking runs with MLflow Set MLflow's own `MLFLOW_TRACKING_URI`, or pass `--mlflow `, to record a PTQ run on an MLflow server so it can be reproduced later from its MLflow entry alone: ```bash python hf_ptq.py \ --pyt_ckpt_path \ --recipe general/ptq/nvfp4_default-kv_fp8_cast \ --export_path \ --mlflow https:/// ``` The run is opened *before* the model loads, so a bad URI or a missing token fails within seconds rather than after a full calibration.
Uploaded artifacts | Artifact | Contents | | --- | --- | | `command.txt` | The full invocation, copy-pasteable, with credentials masked | | `version.txt` | The ModelOpt version that ran | | `experiment.json` | The experiment name, run id and run URL — the same file written into `--export_path` | | `recipe/resolved_recipe.yaml` | The `--recipe` with its `$import`s expanded, so it stands alone | | `logs/hf_ptq.log` | The run's Python stdout/stderr, including the traceback if it crashed | | `summary/quant_summary.txt` | The per-quantizer summary (unless `--no-verbose`) | | `summary/moe.html` | Per-expert calibration token counts, when the run produces them |
Every command-line argument is also logged as a searchable param, alongside `user` / `hostname` / `modelopt_version` / `git_sha` tags. A run that fails is still recorded, with status `FAILED` and its log attached. A tracked run also drops `.experiment.json` into `--export_path`, so a checkpoint found on disk names the run that produced it: ```bash cat /.experiment.json # {"tracking_uri": ..., "experiment_name": ..., "experiment_id": ..., "run_id": ..., # "run_name": ..., "run_url": ...} ``` The local file is written only once the export itself completes, so a run that fails earlier leaves whatever checkpoint is already in `--export_path` — and its pointer — untouched. The `experiment.json` artifact is uploaded for every run that opened, so a failed run stays traceable from the server. An export that is *not* tracked removes any pointer it would otherwise inherit, from a reused `--export_path` or from a tracked source checkpoint. Other flags: - `--mlflow_experiment` — defaults to `$USER/hf_ptq/-`, falling back to `--qformat` when no `--recipe` is used. - `--mlflow_run_name` — defaults to the UTC start time, `YYYYmmdd-HHMMSS`. - `$MLFLOW_TRACKING_URI` enables tracking on its own; `--mlflow` overrides it. A URI taken from the environment is best-effort — if the client is missing or the server is unreachable the run warns and continues untracked, since the variable is often exported for other tooling. An explicit `--mlflow` fails loudly instead. Authentication uses MLflow's own environment variables (`MLFLOW_TRACKING_TOKEN`, or `MLFLOW_TRACKING_USERNAME` / `MLFLOW_TRACKING_PASSWORD`). The tracking itself lives in `modelopt.torch.utils.mlflow` ([`MlflowRunLogger`](../../modelopt/torch/utils/mlflow.py)), so other example scripts can record runs the same way; `hf_ptq.py` only supplies the params and artifacts specific to PTQ. > Note: only the main rank uploads, so `--use_fsdp2` runs produce a single run. The log > captures Python output; output written directly by native libraries (NCCL, CUDA) goes to > the terminal only. On SLURM, keep the job's own `.out` file for those. ## Resources - 📅 [Roadmap](https://github.com/NVIDIA/Model-Optimizer/issues/1699) - 📖 [Documentation](https://nvidia.github.io/Model-Optimizer) - 🎯 [Benchmarks](../benchmark.md) - 💡 [Release Notes](https://nvidia.github.io/Model-Optimizer/reference/0_changelog.html) - 🐛 [File a bug](https://github.com/NVIDIA/Model-Optimizer/issues/new?template=1_bug_report.md) - ✨ [File a Feature Request](https://github.com/NVIDIA/Model-Optimizer/issues/new?template=2_feature_request.md) ### Technical Resources There are many quantization schemes supported in the example scripts: 1. The [FP8 format](https://developer.nvidia.com/blog/nvidia-arm-and-intel-publish-fp8-specification-for-standardization-as-an-interchange-format-for-ai/) is available on the Hopper and Ada GPUs with [CUDA compute capability](https://developer.nvidia.com/cuda-gpus) greater than or equal to 8.9. 1. The [INT8 SmoothQuant](https://arxiv.org/abs/2211.10438), developed by MIT HAN Lab and NVIDIA, is designed to reduce both the GPU memory footprint and inference latency of LLM inference. 1. The [INT4 AWQ](https://arxiv.org/abs/2306.00978) is an INT4 weight only quantization and calibration method. INT4 AWQ is particularly effective for low batch inference where inference latency is dominated by weight loading time rather than the computation time itself. For low batch inference, INT4 AWQ could give lower latency than FP8/INT8 and lower accuracy degradation than INT8. 1. The W4A8 AWQ is an extension of the INT4 AWQ quantization that it also uses FP8 for activation for more speed up and acceleration. 1. The [NVFP4](https://blogs.nvidia.com/blog/generative-ai-studio-ces-geforce-rtx-50-series/) is one of the new FP4 formats supported by NVIDIA Blackwell GPU and demonstrates good accuracy compared with other 4-bit alternatives. NVFP4 can be applied to both model weights as well as activations, providing the potential for both a significant increase in math throughput and reductions in memory footprint and memory bandwidth usage compared to the FP8 data format on Blackwell. For higher accuracy with NVFP4 PTQ, we recommend `nvfp4_mlp_only`, `nvfp4_experts_only`, or `nvfp4_omlp_only`. `nvfp4_mlp_only` restricts NVFP4 quantization to MLP (and MoE) layers only, leaving attention layers in higher precision. `nvfp4_experts_only` quantizes only expert layers (`*mlp.experts*` and `*block_sparse_moe*`), ideal for MoE models. `nvfp4_omlp_only` extends MLP-only by also quantizing the `o_proj` layer, providing a middle ground between full NVFP4 and MLP-only quantization.