NVFP4 Explained: NVIDIA’s 4-Bit Format for Faster AI Inference

NVFP4 is NVIDIA’s 4-bit floating-point format for AI workloads on Blackwell GPUs. It stores tensor values in FP4 E2M1, applies an FP8 E4M3 scale to each block of 16 values, and uses an additional FP32 scale for the whole tensor. This hierarchical scaling makes 4-bit weights and activations more accurate than a simple FP4 conversion while reducing memory traffic and enabling Blackwell Tensor Cores to perform native low-precision matrix multiplication.

For developers, the practical result is potentially higher inference throughput, lower model memory, and better performance per watt. It is not a universal “make my model faster” switch: NVFP4 requires compatible Blackwell hardware, supported kernels, a suitable quantized checkpoint, and accuracy testing on the workload that matters to you.

NVFP4 at a Glance

Why AI Inference Uses Lower Precision

Most of the expensive work in a transformer is matrix multiplication. Model weights and activations must be read from memory, moved through the GPU, and multiplied repeatedly.

Reducing numerical precision helps in two ways:

• fewer bits reduce the memory required to store and move tensors
• hardware can perform more low-precision operations in the same time and power budget

The tradeoff is quantization error. FP16 and BF16 preserve a wide range of values. FP8 keeps fewer. A raw four-bit format has only a small set of representable values, so converting an entire tensor with one scale can erase useful differences between values.

NVFP4’s design is an attempt to capture most of the efficiency of four-bit arithmetic without accepting the accuracy loss of a coarse, single-scale conversion.

How NVFP4 Represents a Value

The four-bit element uses the E2M1 layout:

• 1 sign bit
• 2 exponent bits
• 1 mantissa bit

On its own, E2M1 can represent only a small set of values between approximately -6 and +6. NVFP4 recovers useful tensor range by multiplying the stored value by two scales:

reconstructed_value = fp4_value × block_scale × global_scale

The pieces are:

• fp4_value: the E2M1 value stored in four bits
• block_scale: an FP8 E4M3 scale shared by 16 nearby values
• global_scale: an FP32 scale shared by the tensor

Think of the FP4 value as the local detail, the block scale as a local zoom level, and the global scale as the tensor-wide zoom level.

This distinction matters. Saying “NVFP4 uses four bits” is directionally correct, but incomplete. NVIDIA calculates an effective storage cost of about 4.5 bits per value before the tiny per-tensor overhead because every 16 values also need one eight-bit block scale.

Why Blocks of 16 Improve Accuracy

Tensor values are not distributed evenly. One part of a weight matrix may contain small values, while a nearby region contains outliers.

If hundreds or thousands of values share one scale, an outlier can stretch the range and force many ordinary values into the same few quantized buckets. NVFP4 recomputes the scale every 16 elements, so each small region adapts to its own range.

The smaller block also explains an important difference between NVFP4 and MXFP4:

• NVFP4 uses one scale per 16 values
• MXFP4 uses one scale per 32 values

Smaller blocks isolate outliers more effectively, although the additional scales create some storage and implementation overhead.

Why the Block Scale Uses FP8 E4M3

MXFP4 uses an E8M0 scale. E8M0 covers a wide range but represents scales as powers of two, so scale selection is coarse.

NVFP4 uses FP8 E4M3 for block scales. Its mantissa can represent fractional scale values, allowing a scale to fit the local data more closely. The tradeoff is that E4M3 has less range than E8M0. NVFP4 compensates with the second, tensor-wide FP32 scale.

This two-level design gives NVFP4:

• fine local adaptation through FP8 E4M3 block scales
• broad tensor range through the FP32 global scale
• compact E2M1 storage for the actual values

The format’s accuracy comes from the combination, not from E2M1 alone.

NVFP4 vs FP8, MXFP4, and INT4

NVFP4 is not automatically better than every INT4 or FP8 recipe. A well-tuned weight-only INT4 model may run on more hardware. FP8 may be easier to deploy and more tolerant of sensitive layers. NVFP4 becomes compelling when the model, kernels, and Blackwell hardware can use W4A4 computation efficiently.

What W4A4, W4A8, and W4A16 Mean

Quantization names often describe the precision of weights and activations:

• W4A4: four-bit weights and four-bit activations
• W4A8: four-bit weights and eight-bit activations
• W4A16: four-bit weights and 16-bit activations

Compressing weights reduces checkpoint and memory-bandwidth pressure. Quantizing activations as well enables more of the matrix multiplication to run through native FP4 Tensor Core paths.

This is why two “NVFP4 models” can perform differently. One may use W4A4 for most linear layers, while another stores FP4 weights but falls back to higher-precision activations or kernels. Always inspect the quantization recipe and runtime path, not only the checkpoint name.

Memory and Accuracy: What the Published Results Show

NVIDIA reports that NVFP4 uses approximately:

• 3.5x less model memory than FP16
• 1.8x less model memory than FP8

The result is below the theoretical 4x and 2x reductions because block scales and metadata take space.

For accuracy, NVIDIA compared FP8 and post-training-quantized NVFP4 versions of DeepSeek-R1-0528 across seven evaluations. The reported differences were generally one percentage point or less, with the NVFP4 version scoring higher on one of the tests.

Those results show that NVFP4 can retain strong model quality. They do not prove that every model, domain, or quantization recipe will remain within one point. Calibration data, outliers, architecture, sensitive layers, KV-cache precision, and evaluation design all matter.

What Unsloth Dynamic NVFP4 Adds

NVFP4 is the numerical format. Unsloth Dynamic NVFP4 is a model-quantization recipe built around that format.

Instead of forcing every layer into W4A4, Unsloth analyzes the model and leaves accuracy-sensitive layers in FP8 W8A8 or BF16. The remaining layers use NVFP4 W4A4 so they can benefit from Blackwell FP4 Tensor Cores.

This is a common low-precision strategy: spend higher precision where it protects accuracy and use lower precision where it saves the most memory and compute.

Unsloth reports model-specific speedups of up to 2.5x for its current Qwen3.6 Dynamic NVFP4 checkpoints, along with calibrated FP8 KV caches for longer context capacity. Those numbers are specific to its checkpoints, kernels, B200 tests, and concurrency settings. Treat them as evidence to benchmark, not a universal NVFP4 multiplier.

Hardware and Software Requirements

Native NVFP4 execution requires NVIDIA Blackwell hardware. Depending on the framework and kernel support, that includes data-center and workstation or consumer Blackwell products such as:

• B200 and B300
• GB200 or GB300 systems
• RTX PRO Blackwell GPUs
• GeForce RTX 50-series GPUs
• DGX Spark

The software path matters just as much. Current options include:

• NVIDIA Model Optimizer for post-training quantization and quantization-aware training
• Transformer Engine for NVFP4 training recipes
• TensorRT-LLM for optimized deployment
• vLLM and SGLang for supported prequantized checkpoints
• Hugging Face checkpoints exported in compatible formats

An NVFP4 checkpoint running through an unsupported fallback kernel may be slower than expected. Framework version, CUDA version, GPU compute capability, model architecture, and backend selection all need to align.

Running a Prequantized NVFP4 Model

The fastest way to experiment is usually a published checkpoint that already includes the quantization metadata.

Unsloth currently documents serving its Qwen3.6 NVFP4 models with vLLM:

uv venv unsloth-nvfp4-env --python 3.13
source unsloth-nvfp4-env/bin/activate

uv pip install "vllm>=0.25.0" \
  "flashinfer-python>=0.6.13" \
  "nvidia-cutlass-dsl>=4.5.2" \
  --torch-backend=auto

vllm serve unsloth/Qwen3.6-27B-NVFP4

Use versions recommended by the checkpoint publisher, because low-precision kernel support changes quickly. Unsloth also recommends allowing vLLM to choose the backend in normal deployments; forcing an unsuitable backend can remove the W4A4 performance advantage.

Before exposing the server to production traffic, verify that logs show the expected NVFP4 kernels rather than a dequantized or W4A16 fallback path.

Quantizing Your Own Model

A production post-training quantization workflow normally looks like this:

1. Start with a supported BF16, FP16, or FP8 checkpoint.
2. Choose a representative calibration dataset.
3. Apply an NVFP4 recipe using NVIDIA Model Optimizer or another supported tool.
4. Keep sensitive modules at higher precision if evaluation quality drops.
5. Export a unified checkpoint for the target inference engine.
6. Build or load the engine on Blackwell hardware.
7. Compare quality, latency, throughput, memory, and power against the original model.

Calibration data should resemble real prompts and outputs. A generic text sample may preserve perplexity while damaging tool calls, structured output, code generation, domain terminology, multilingual behavior, or long-context retrieval.

For a customer-facing model, create an evaluation set before quantizing. Without a baseline, “the model seems fine” is not a defensible quality check.

How to Benchmark NVFP4 Correctly

Measure the complete inference workload, not only peak Tensor Core FLOPS.

Track:

Use identical prompts, sampling settings, context lengths, output limits, batch sizes, and concurrency. Warm up each engine, report software versions, and confirm that both runs use the intended kernels.

When NVFP4 Is a Good Fit

NVFP4 is worth testing when:

• production inference already targets Blackwell GPUs
• model weights and activations are a memory or bandwidth bottleneck
• higher throughput or larger batches have clear business value
• a compatible prequantized checkpoint exists
• the team can run representative accuracy evaluations
• the runtime exposes native W4A4 kernels for the model

It is especially attractive for large language models, mixture-of-experts systems, reasoning workloads, and multimodal models where memory movement dominates cost.

When NVFP4 Is Not the Right First Choice

Choose another path when:

• deployment must run on Hopper, Ampere, AMD GPUs, Apple Silicon, or CPUs
• ecosystem portability matters more than peak Blackwell performance
• the model is small enough that quantization is not the bottleneck
• critical quality cannot tolerate the observed accuracy change
• the serving stack silently falls back to slower kernels
• the team does not yet have a reliable evaluation suite

FP8 is often a simpler step for Hopper and Blackwell fleets. Weight-only INT4 or GGUF formats may be more practical for broad local deployment. The right format is the one that improves the real system under its actual hardware and quality constraints.

Production Checklist

Before approving an NVFP4 deployment:

• confirm the exact GPU architecture and compute capability
• pin CUDA, framework, inference-engine, and kernel versions
• verify the checkpoint’s weight and activation precision
• inspect logs for native NVFP4 kernel selection
• evaluate domain accuracy, tool calls, structured output, and safety behavior
• test short and long contexts
• compare time to first token and inter-token latency
• benchmark single-user and production concurrency
• measure peak memory and batch capacity
• keep an FP8 or BF16 rollback path

Quantization is a systems optimization. Model format, kernel, GPU, scheduler, KV cache, batching, and evaluation all contribute to the outcome.

FAQ

What does NVFP4 stand for?

NVFP4 refers to NVIDIA’s FP4 format and scaling recipe for Blackwell AI workloads. Tensor elements use a four-bit E2M1 floating-point representation combined with block-level and tensor-level scales.

Is NVFP4 exactly four bits per parameter?

The tensor element is four bits, but each block of 16 elements also stores an eight-bit scale, plus one global FP32 scale per tensor. NVIDIA describes the resulting element-and-block-scale cost as approximately 4.5 bits per value.

What is the difference between NVFP4 and MXFP4?

Both use E2M1 four-bit values. NVFP4 uses an FP8 E4M3 scale for every 16 values plus a global FP32 scale. MXFP4 uses an E8M0 power-of-two scale for every 32 values. NVFP4’s smaller blocks and fractional scales generally provide finer local adaptation.

Does NVFP4 work on H100 or H200 GPUs?

No native NVFP4 Tensor Core path is available on Hopper H100 or H200 GPUs. Those GPUs have strong FP8 support, which is usually the appropriate low-precision format for that generation. NVFP4 targets Blackwell.

Can NVFP4 quantize both weights and activations?

Yes. NVFP4 supports low-precision weights and dynamically quantized activations. Specific checkpoints may keep some activations or sensitive layers in FP8 or BF16, so inspect the actual recipe.

Is NVFP4 always faster than FP8?

No. NVFP4 reduces data movement and can unlock faster W4A4 Tensor Core operations, but end-to-end performance depends on model shape, kernels, batching, concurrency, KV-cache behavior, and fallback operations. Benchmark the target deployment.

How much accuracy does NVFP4 lose?

There is no universal number. NVIDIA reported differences of roughly one percentage point or less on its DeepSeek-R1-0528 comparison, while other models and tasks can behave differently. Representative calibration and task-specific evaluation are required.

Is Unsloth Dynamic NVFP4 the same as NVIDIA NVFP4?

It uses NVIDIA’s NVFP4 format, but adds a mixed-precision selection recipe. Unsloth keeps important layers in FP8 or BF16 and quantizes other layers to W4A4 to balance speed and accuracy.

Sources and Further Reading

The format details, hardware requirements, benchmark figures, and deployment guidance in this article were checked against these primary and implementation sources:

• Introducing NVFP4 for Efficient and Accurate Low-Precision Inference — NVIDIA’s format overview, DeepSeek-R1 accuracy comparison, and memory analysis
• NVFP4 in NVIDIA Transformer Engine — E2M1 representation, hierarchical scaling, training behavior, and supported devices
• TensorRT Quantization Schemes — formal FP4 quantization behavior and block-size requirements
• TensorRT-LLM Quantization Guide — runtime, model, hardware, and KV-cache support
• Unsloth Dynamic NVFP4 Guide — mixed-precision layer selection, current checkpoints, vLLM commands, and vendor benchmarks
• Microscaling Data Formats for Deep Learning — background research on block-scaled low-precision formats

Software support and benchmark results change quickly. Check the current documentation for your GPU, checkpoint, inference engine, and kernel versions before making a production decision.

Final Take

NVFP4 matters because it makes four-bit inference a hardware-and-software system rather than a blunt storage conversion.

The E2M1 values provide compact computation. FP8 scales adapt every 16 values to local tensor behavior. The global FP32 scale preserves overall range. Mixed-precision recipes can protect the layers that do not tolerate four-bit quantization.

On compatible Blackwell hardware, that combination can reduce model memory, increase throughput, and preserve much of the quality of higher-precision inference. The right adoption path is simple: start with a supported checkpoint, verify native kernels, benchmark the complete serving workload, and let task-specific evaluation—not the format name—decide whether it is ready for production.