SkyRL Adopts FP8 for RL, Cuts Rollout Time by 23%
SkyRL, a reinforcement learning (RL) framework, has introduced FP8 precision across its training and rollout processes, enabling faster model convergence with reduced hardware overhead. According to a blog post published on August 25, 2026, FP8 cuts RL step times by up to 23% on NVIDIA Hopper (H100) and Blackwell (B200) GPUs, while maintaining performance on par with BF16. This advancement signals a shift toward precision optimization in RL workloads, particularly for large language models (LLMs) and RLHF (Reinforcement Learning with Human Feedback).
The innovation hinges on SkyRL’s on-policy weight synchronization (OPWS), which ensures that the rollout engine operates with the exact FP8-trained policy, avoiding numerical mismatch that previously plagued independent FP8 configurations. This refinement addresses a challenge noted in other FP8 RL stacks, such as NVIDIA's NeMo RL and recent academic proposals like Jet-RL, which warned against instability caused by mismatched quantization between training and rollout phases.
FP8’s Role in RL Efficiency
FP8, or 8-bit floating-point precision, offers significant advantages over BF16 and FP32 in RL workloads. It reduces memory traffic, lowers GPU memory requirements, and increases arithmetic throughput on tensor cores. For workloads like rollout generation—which dominates RL step time—FP8 translates directly into faster decoding and higher efficiency. SkyRL reports that FP8 parameter storage reduces per-GPU memory usage by 39-42%, freeing up resources for larger models or longer-context rollouts.
These optimizations are particularly impactful for LLM reinforcement learning, where memory bottlenecks often limit performance. For example, SkyRL’s FP8 stack was tested with the Qwen3.5 model family, including the dense 9B and mixture-of-experts (MoE) 35B-A3B configurations, achieving step-time reductions of 19% to 23% in long-rollout scenarios.
Addressing Stability: Why OPWS Matters
One of the primary risks in FP8 reinforcement learning is instability from inconsistent quantization between the training engine and the rollout engine. SkyRL’s OPWS eliminates this issue by directly transferring trainer-produced FP8 weights and block-scale metadata to the rollout engine, avoiding intermediate dequantization and re-quantization steps. This ensures numerical consistency, which is crucial for on-policy RL where the sampled trajectories must align with the policy being optimized.
In tests with the Qwen3.5-9B model on eight H100 GPUs, SkyRL demonstrated that OPWS kept the train-rollout log-probability gap stable at 0.02 to 0.03 across 400 steps—well within acceptable bounds for RL workloads. By contrast, older methods with independent quantization showed divergence, with log-probability gaps exceeding 0.05 under similar conditions.
Implications for the Industry
The adoption of FP8 precision in SkyRL underscores a broader industry trend toward low-precision AI workloads. NVIDIA has been a driving force in this area, with its April 2026 blog highlighting FP8's ability to accelerate RL while preserving accuracy. Academic work, such as the FP8-RL paper published in January 2026, has also explored practical FP8 implementations for RL stacks like veRL and Megatron-LM.
While FP8 reinforcement learning is not a standalone market, its commercial relevance ties directly to AI hardware and software ecosystems. NVIDIA’s Hopper and Blackwell GPUs, along with frameworks like NeMo RL and SkyRL, are leading platforms for deploying FP8 at scale. By reducing the cost and complexity of RL fine-tuning, FP8 could make advanced LLM optimization accessible to smaller players, further democratizing AI development.
What’s Next?
SkyRL’s FP8 stack is already proving its value in efficiency and scalability, but challenges remain. FP8 does not yet cover every operation—attention mechanisms and optimizer states still rely on higher precision. Additionally, hardware-specific constraints, such as Blackwell's reliance on coarse-grained quantization, may limit adoption in certain configurations.
Nonetheless, SkyRL’s success with FP8 reinforces its potential to reshape reinforcement learning workflows. With growing interest from researchers and industry leaders, FP8 seems poised to become a cornerstone of RL optimization on next-generation AI accelerators.