Introduction

RL Training Simulator. The simulator models an asynchronous RL system where samplers continuously generate rollouts while the trainer updates the policy. Policy mismatch arises from both off-policyness (stale rollouts) and quantization error, accumulating into policy drift that eventually degrades reward if it exceeds the optimizer's correction capacity.

RL & Efficiency Knobs. Off-policy, weight sync, batch size, and horizon control the degree of asynchrony and policy staleness. MXFP8 and NVFP4 improve training and rollout efficiency, while dequantized backward, BF16 last 15%, and shared experts improve numerical stability.

How to use it. Explore the throughput–stability tradeoff by varying algorithmic and systems knobs. Increase asynchrony or lower precision to improve utilization, then observe how stabilization techniques recover the stability margin. The goal is to identify configurations that maximize throughput while keeping policy mismatch below the critical threshold where drift accelerates and reward collapses.

The essence of RL is to teach a model through action and consequence; at humans&, we use RL to train models that understand the long-term impacts of their interactions with people. In an RL loop1Specifically a policy gradient algorithm., a model acts, gets a reward, then updates the actions' likelihoods. However, in real-world RL training for LMs, we want to update the policy from observations as soon as they're available, even as other rollouts are still being sampled. For the long-horizon multiplayer rollouts central to our mission, we may even take dozens of training steps as a model completes one rollout.

This introduces the tug-of-war between throughput and stability in RL. On one hand, we want to train on as many examples as quickly as possible2Make each training step faster, each rollout faster, or overlap trainers and samplers more.. On the other hand, most techniques to increase throughput cause the sampled policy and the trained policy to diverge, potentially slowing learning and destabilizing training. Quantization exemplifies this tradeoff: while low precision formats enable faster communication and computation on hardware, quantization hurts stability. Fast and accurate low precision formats like NVFP4 paired with hardware support3Allowing up to 9x more operations per second than 16-bit training on NVIDIA Rubin GPUs. Dense Tensor Core PFLOPS per GPU from NVIDIA's HGX platform specs (B200/B300 from 8-GPU HGX systems, Rubin from the HGX Rubin NVL8 table):GPUBF16FP8FP4B2002.254.59B3002.254.513.5Rubin (NVL8)417.535 have driven large throughput increases in both model training and inference separately.

However, there is no stable, hardware-native 4-bit4While INT4 QAT recipes exist to simulate 4-bit quantization, these have used 16-bit activations (Kimi K2 Thinking), and MXFP4 recipes like in DSv4 have used MXFP8 activations (DeepSeek-V4 on Day 0). Our approach uses NVFP4 quantization for both weights and activations. RL recipe in the open-source community, largely because the sampling and training instabilities compound in RL. In a long-running collaboration with the open source community, we have developed and shared a low-precision RL recipe preserving higher-precision training dynamics. In this recipe, we needed to address instability from the forward pass due to policy quantization errors, from the backward pass due to gradient mismatches, and at the intersection of both due to a small set of particularly sensitive weights. Below, we explain how we addressed each and validated the final recipe.

Note this effort could not have happened without our amazing collaborators at RadixArk and NVIDIA, and their work across the training, inference, and RL stacks.

Baseline: A starting recipe with stable training dynamics

For consistent comparison, unless otherwise noted, all experiments in this report use the Qwen3-30B-A3B model trained with 8k sequence length on DAPO-math-17k datasets.

We start with a baseline recipe that uses the NVFP4 format. NVFP4 is a 4-bit floating-point format that improves throughput and memory efficiency on newer NVIDIA GPU architectures. It uses hierarchical block scaling: each block of 16 values has an FP8 E4M3 scale while the full tensor has a global FP32 scale. Together, these scales can recover a higher-precision value.

Why is the NVFP4 pretraining recipe not sufficient for RL?

NVIDIA's NVFP4 pretraining recipe is the natural starting point for this work. It uses a mixed-precision strategy, in which most operations are performed in FP4 precision, while numerically sensitive components remain in higher precision.

However, pretraining and RL have different failure modes.

In pretraining, the gradient signal is dense and repeatedly averaged over a large number of tokens. A main goal is to avoid quantization bias that would perturb the direction of optimization. Stochastic rounding helps by making the gradient quantization approximately unbiased.

The RL setting has a different bias-variance tradeoff. The policy-gradient is already a noisy estimator because it depends on the sampled rollouts, advantage estimation, reward estimates, KL regularization, and policy staleness. Therefore, it is not sufficient for the quantization method to be just unbiased. Quantization noise must be small enough that it does not degrade the per-update true policy gradient signal.

Therefore, we prioritize interventions that decrease quantization error in the policy and improve the accuracy of the gradient computation.

Baseline recipe

For the baseline recipe, we quantize only the MoE layers and keep all other layers in higher-precision BF16. In DeepSeek-V3-style architectures, MoE experts account for 97%5Each expert contains 3 projection matrices of size 7168 x 2048, giving about 44M parameters per expert. With 256 experts and 1 additional shared expert per MOE layer, that's about 11.32B parameters per layer. Across 58 layers, that's 656B parameters, which is approximately 97.8% of total parameters. of the total parameters so aggressively quantizing MoE layers results in most of the memory benefits.

We run the forward pass in NVFP4, while keeping the backward pass in BF16 precision. This is a conservative starting point to give us rollout and memory benefits of FP4, while avoiding FP4 in the backward pass.

For weights, we use the standard NVFP4 format with FP8 per-block scales and a single FP32 global scale.

For activations, we do not use a global FP32 scale computed over the entire tensor. As the Cursor Composer 2 technical report points out, the global scale can create two issues:

• The same token can be quantized differently depending on which other tokens are present in the batch.
• If the tensor contains multiple positions from the same sequence, a later token can influence the shared scale used by an earlier token, creating a leakage path from future tokens to past tokens.

To solve this problem, similar to Cursor Composer 2, we use per-token activation scaling where each token computes its own FP32 activation scale across the hidden dimension. This keeps quantization local to each token, and avoids a separate calibration step, since scales can be computed directly from the activations observed during the forward pass. Also, this fine-grained FP32 scale helps with the lower quantization error compared to the per-tensor FP32 scale.

During rollout, this per-token scale computation is fused into the activation quantization kernel. For each token row, the kernel computes the FP32 scale, the FP8 E4M3 scales for 16-value blocks, and packs these quantized values into FP4. This fused implementation reduces extra memory movement and kernel launch overhead.

Implementation

The per-token NVFP4 recipe required changes across the stack and we have open-sourced our implementation:

• TransformerEngine row scaled recipe
• cuDNN kernel for row-scaled grouped GEMM quantization
• FlashInfer inference kernel for MoE per-token quantization
• SGLang support for per-token NVFP4 MoE

Improving Gradient Stability

We observed gradient norm spikes during training with our open-source per-token NVFP4 recipe. This recipe is intentionally conservative with precision in backward pass: although the forward pass uses NVFP4 operands, the backward pass keeps weights and activations in BF16 precision. This is much more stable than running the full backward path through coarse-grained NVFP4, but it introduces a mismatch between forward and backward pass which causes occasional gradient spikes.

The issue is that the backward pass is no longer differentiating the same function that was used in the forward pass. For example, in a simple linear layer, the forward pass evaluates y = x · Q(w_bf16), where Q(.) denotes quantization6Quantization function behaves like a staircase function that's not differentiable at step change values. Quantization usually involves clipping and rounding i.e clip(round(w/s), qmin, qmax).. The backward pass behaves as if it is differentiating y = x · w_bf16. This decision makes backward pass more stable, but also means that the backward pass is unaware of the clipping and rounding decisions made in the quantization function.

To mitigate this chain-rule inconsistency while preserving the stability of the BF16 backward pass, we use dequantized backward. Instead of the BF16 precision in the backward pass, we use the BF16-dequantized value of the exact quantized tensor used in the forward pass. In case of the linear layer example, that means y = x · DQ(Q(w_bf16)), where DQ denotes dequantization.

Here, the backward pass still uses BF16 operands, but the operands reflect the same NVFP4 quantization decisions that were used in the forward pass. Note that our recipe applies dequantized backward operation on both weights and activations.

Results

Validation with MXFP8:

To validate our changes, we first tested this fix with the MXFP8 recipe. The figure below shows how the gradient norm behaves under 3 different recipes:

• a) quantized forward and backward, shown as mxfp8-mxfp8
• b) quantized forward and BF16 backward, shown as mxfp8-high-precision-bf16
• c) quantized forward with the chain-rule violation corrected using dequantized weights in the backward pass, shown as mxfp8-dequantized-bf16

Our method leads to cleaner and less noisy gradients compared with MXFP8 backward throughout all stages of the training. Compared with high-precision BF16 backward, our method helps prevent the gradient norm from increasing and is consistently stable.

Validation with NVFP4:

In comparison to MXFP8, NVFP4 complicates training dynamics. On one hand, our method helps stability by providing less biased gradients because the backward pass matches the quantized forward pass. On the other hand, NVFP4 quantization can increase gradient variance due to its coarser representation grid.

To understand this empirically, our first setup uses SGD optimizer instead of Adam because SGD makes it easier to isolate our method's effect on the true gradients.

We observe that our method leads to faster learning than the higher-precision backward baseline. This can be explained by the fact that the gradient is better aligned with the actual quantized function being trained.

However, we also observed that this recipe is less stable and can diverge. As discussed earlier, this is likely because the gradients can have higher variance due to the coarser NVFP4 representation.

To replicate the setup of our real RL runs, we switch back to Adam optimizer and check whether properties such as momentum and adaptive second-moment reduce the additional variance introduced by our method.

We observe that our method produces far fewer large gradient spikes than the baseline method that uses higher precision in the backward pass.

Implementation

We open-sourced this feature in TransformerEngine. It allows users to easily switch between quantized backward, regular BF16 backward precision, and our method, which uses dequantized weights in the backward pass.

Additionally, in collaboration with NVIDIA, we reduced the dequantization overhead during training. Our TransformerEngine implementation avoids storing additional quantized tensor copies, which are stored by default in lower-precision training. This reduces peak memory consumption during training by 70%7NVFP4 linear peak memory as measured in TE implementation for a proxy workload (input: 2k x 2k, output features 8k, dtype=bf16) reduces from 150MB to 45MB..

Four-Over-Six for RL Weights and Activations

A further challenge has to do with NVFP4 itself. Typically, we can represent parameters as multiples of 0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, ±6. However, this means that when representing a value near 5/6 of the range, the error can be as much as 1/6 of the range! If we instead map the maximal values to ±4, the largest possible error becomes 1/8 of the range. This approach, known as four-over-six, or 4/6, comes from the paper Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling. Critically, the reduced range is only introduced when it decreases the quantization error. In the paper, this technique was applied only to the pretraining and only for the activations; however, we apply it to the weights as well. This matters more for RL because we start from already-trained weights and errors in their quantization have disproportionate impacts8We add support for 4/6 in either weights and activations or both..

For the adaptive scaling, we need to compute the adjusted weights and determine per block whether the error decreased.

However, the reference and most straightforward method to compute the adjusted weights9As in the reference implementation. is expensive, i.e.:

candidate = fp4_value * fp8_scale * amax / (E2M1_MAXVAL * E4M3_4OVER6_MAXVAL)

To choose which maximum FP4 value we want (4 or 6), we

• convert each stored FP4 value to FP32
• multiply it with the block scaling value and the tensor/token maximum
• normalize by dividing by the maximum FP4 / FP8 values
• compute the blockwise quantization error for both 4 and 6
• compare the block quantization error10In mean-squared error (MSE) or mean-absolute error (MAE). to choose 4 or 6.

The problem is that all of these operations are scalar FP32 arithmetic, which has limited throughput on current hardware. This leads to stalls.

We instead perform the data type casting and FP4*FP8 computation losslessly in FP1611Notably not in BF16, as BF16 does not have enough mantissa bits to represent all E2M1 x E4M3 products precisely. using hardware-native PTX operations12For casting, cvt.rn.f16x2.e2m1x2 and cvt.rn.f16x2.e4m3x2. For E4M3 x E2M1 dequantization, mul.rn.f16x2.. In addition, instead of scaling up each candidate using amax / (E2M1_MAXVAL * E4M3_4OVER6_MAXVAL) twice per-element, we scale the target down by its inverse, which allows us to keep that computation in FP16. We could, of course, make this even faster by lossily approximating the error - instead, we keep the error accumulation in FP32, in the interest of high stability and we do not observe kernel speedups when switching to BF16 or FP16. We found that across the board, the resulting scaling factor chosen matched over 99.97% of the time in a quantization benchmark, regardless of the type of error used or the quantization mode, while providing a ~2.8x speedup in practice over the naive/strict approach. Lastly, to avoid any trainer-sampler mismatch, we needed a contract between both sides of the RL stack; we ensured that our 4/6 implementation was a bit-exact match across both training and sampling.

For this bit-exact contract, we need to ensure that the quantized values and scales are bitwise identical across the trainer and the sampler every single time. First, our kernels need to produce bitwise deterministic output every time. Second, the implemented kernels for training and inference need to follow the exact same numerical contract.

For 4/6 this also needs to be true for the error computation, accumulation, and comparison. Reductions for error accumulation were particularly hard, given that consistency needed to be maintained regardless of weight layout and swizzling.

As seen in Figure 15, adding 4/6 does not noticeably increase rollout time. The added operations are fused into the quantization kernel, hiding the extra compute work under existing memory access latencies.

We implemented this contract across FlashInfer and TransformerEngine, two kernel libraries maintained by NVIDIA.

As seen in Figure 17, adding 4/6 meaningfully stabilizes reference KL compared with NVFP4 RL without 4/6.

Implementation

We open-sourced the full bit-exact 4/6 implementation across training and inference stack, together with the optimized FP16 fast path:

• TransformerEngine 4/6 recipe
• TransformerEngine 4/6 optimized FP16 contract
• FlashInfer 4/6 recipe
• FlashInfer 4/6 FP16 contract
• FlashInfer CuTe DSL 4/6 MoE kernels

Selective Layer Precisions: Spending Where It Matters

In practice, no low-precision recipe uses the same precision for every weight and activation. As in the NVIDIA NVFP4 pretraining paper, we focus on two sets of weights: first, we keep a small fraction of final layers in high precision13BF16; we generally use 15% as suggested by NVIDIA in the pretraining paper.; second, many MoE architectures include an expert that is always active (a "shared" expert) - keeping this expert in higher precision is relatively inexpensive but impactful since it is active for every token. We visualize the memory vs logprob tradeoff in mxfp8 below; note that the optimal tradeoff here will generally depend on the model and memory available.

Moreover, when we decide that some weight should be in a higher precision, it should maintain that precision in checkpointing, training, rollouts, and weight updates. The naive approach of having bespoke per-model quantization handling in each context is error-prone, so we introduced infrastructure to minimize the complexity.

Implementation

We introduced this infrastructure across the training and serving stack:

• miles#614
• miles#1054
• miles#1261
• sglang#18742

Putting it all together: Final Recipe

After implementing and integrating these optimizations, we need to convince ourselves of the stability of the combined recipe. For the sake of fairness, we did not tune hyperparameters for any of our quantized runs, but used our method as a drop-in replacement for BF16. Even with two out of the three individual optimizations, we still see grad norm spikes:

However, with all three optimizations, the gradient norm is remarkably stable (five runs pictured):

Furthermore, the reward curve for all of these runs closely tracks the BF16 reward curve.

A Pleasant side effect: Online NVFP4 Serving

The same rollout quantization path can serve model checkpoints with online post-training quantization. Operationally, this helps us deploy trained checkpoints efficiently without running calibration, QAT, or separate model-conversion pipelines, and can leverage our quality improvements for quantized models such as 4/6 and selective high precision.

For example, in an internal benchmark we have GLM 5.1 as judge score multi-turn rollouts using a variety of tools. The correlation between the FP8 deployment and our NVFP4 quantized deployment is .924.

We have open-sourced this implementation. Using this feature in SGLang is as simple as adding --quantization nvfp4_online to any existing BF16/FP8 model serving commands.

Final thoughts

Future Improvements

A difficulty of using NVFP4 for training is the live quantization of activations. Finding the scale factor involves either calibration or finding the dynamic per tensor (or per token in our case) maximum and rescaling, which leads to performance overhead due to in-kernel reductions. One solution is to use MXFP8 activations with NVFP4 weights. This is doubly useful because activations tend to be harder to quantize than weights, so this recipe would increase precision where it would be most useful. Existing NVIDIA GPUs ship MXFP8 x MXFP4 instructions, but quantizing weights to MXFP4 decreases precision due to the UE8M0 scaling factors they use. We hope this is available in future hardware generations.

We view our work as co-designing the quantization configuration with the training environment: As we are scaling on ultra long horizon, multi-agent training of much larger models, we also face unique challenges, we choose a quantization configuration that prioritizes stability over fast training. In the future, more parts of the training recipe should be customized for interaction with low precision. In particular, designing optimizers around the unique challenges of low precision training appears to be a promising direction of inquiry.

Our stack is built on open-source code, and we are glad to contribute back to the community.

In the implementations linked, you can find customization options for each feature. Almost all of these can be configured in isolation, as long as the 4/6 arguments for TransformerEngine and FlashInfer line up.

Conclusion

At this point, we would normally take this chance to mostly connect this exact work to the company and its goals. Instead, let us share a metaphor. In this work, some challenges naturally arise from the individual components but the most critical questions come from the interaction between the pieces. It goes without saying that if making RL with NVFP4 work was as straightforward as applying an NVFP4 pretraining recipe to the trainers and NVFP4 inference on samplers, there would be no need to write or open-source anything.

Most researchers have heard of the bitter lesson; a less-discussed-but-equally-thoughtful short essay introduced the “one-step trap”. Modeling complex dynamical systems in terms of local predictions is often a trap. In this work, we mostly focused on the interaction between trainers and samplers; in our company, we focus on the dynamics of the interactions between people and models. If we exclusively build models optimized for standalone tasks, they will never learn the non-local long-horizon value that they can bring by understanding people and their goals and interactions with others. The single-step approach is an easier one, but almost never the one worth taking.

Acknowledgement

We would like to once again thank our amazing collaborators at RadixArk and NVIDIA, who worked with us on integrating these features with FlashInfer, TransformerEngine, SGLang, Megatron-LM and Miles. None of this would be possible without them!

References

• Composer 2 Technical Report, Cursor Research Team.
• Pretraining Large Language Models with NVFP4, NVIDIA.
• Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling, Cook et al.
• mit-han-lab/fouroversix, reference Four Over Six implementation.
• Scalable Training of Mixture-of-Experts Models with Megatron Core, Yan et al.
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale, Yu et al.
• Qwen3 Technical Report, Qwen Team.
• DeepSeek-V3 Technical Report, DeepSeek-AI.
• The Bitter Lesson, Rich Sutton.
• The One-Step Trap (in AI Research), Rich Sutton.

Citation

Please cite this work as:

Li, Ziang and humans& ai, "The 4-bitter Lesson: Balancing Stability and Performance in NVFP4 RL",
humans& ai Blog, Jul 2026.

Or use the BibTeX citation:

@article{li2026nvfp4rl,
  author = {Ziang Li and humans\& ai},
  title = {The 4-bitter Lesson: Balancing Stability and Performance in NVFP4 RL},
  journal = {humans\& ai Blog},
  year = {2026},
  month = jul,
  note = {https://humansand.ai/blog/nvfp4-rl.html}
}