NVIDIA's Vera Rubin NVL72 Achieves Exceptional Performance in MLPerf Inference v6.1

In the realm of AI inference economics, system performance, efficient infrastructure scaling, and continuous software optimization stand out as critical factors. Maximizing system performance translates to generating more tokens, resulting in increased revenue. Efficient scaling is vital as it ensures that throughput rises proportionally when adding hardware, minimizing resource expenditure while serving users effectively. Continuous optimization allows organizations to extract more value from their infrastructure investments. Coupled with platform fungibility, which allows any infrastructure to support various models and workloads, high utilization rates can be maintained.

NVIDIA's platform is engineered to enhance these aspects, as demonstrated by the impressive results from MLPerf Inference v6.1:

• NVIDIA Vera Rubin NVL72 Debuts: This system's inaugural MLPerf Inference submission reveals a throughput improvement of up to 3.7x compared to its predecessor, GB300 NVL72.
• Exceptional Efficiency of NVIDIA GB300 NVL72: A 288-GPU configuration across four GB300 NVL72 racks achieved an impressive 99% scaling efficiency, showcasing nearly linear throughput growth from a single-rack baseline.
• Software Optimizations Yield Performance Gains: Continuous software enhancements contributed to a performance boost of up to 1.6x over the previous version, v6.0.

Vera Rubin NVL72 Achieves Remarkable MLPerf Inference Results

The Vera Rubin NVL72 system was evaluated against some of the most challenging benchmarks in the MLPerf Inference v6.1 suite, specifically focusing on DeepSeek-R1 and Qwen3-VL. Impressively, the Vera Rubin NVL72 achieved 3.7x higher throughput than the GB300 NVL72 on the Qwen3-VL benchmark across offline, server, and interactive scenarios, leveraging the vLLM alongside NVIDIA's Dynamo open-source inference framework. The DeepSeek-R1 benchmark showed that throughput reached up to 2.5x that of GB300 NVL72 when using the NVIDIA TensorRT-LLM library. Such promising early results underscore NVIDIA's commitment to rapid innovation and highlight the expected continual enhancement in performance due to ongoing software optimizations.

The implications of this performance are significant; every Vera Rubin NVL72 rack can produce many more tokens, service a larger number of users, and ultimately yield higher revenue compared to a GB300 NVL72 rack, all while decreasing the cost per token. This achievement stemmed from a comprehensive approach to codesign between hardware and software. The advanced Tensor Cores and Transformer Engine significantly accelerated both the prefill and decode phases of inference, while NVFP4 precision reduced memory requirements for model weights, attention, and KV cache, allowing for increased throughput with minimal degradation of output quality.

Moreover, Vera Rubin's submissions incorporated disaggregated serving processes, effectively separating prefill and decode along with large-scale expert parallelism. This method maximized efficiency across the mixture-of-experts layers vital for models like DeepSeek-R1 and Qwen3-VL. The innovative NVL72 scale-up domain, enhanced by sixth-generation NVIDIA NVLink and NVLink Switch, delivers 10x higher packet rates and three times lower latency compared to standard Ethernet, thereby providing a robust interconnect foundation necessary for these strategies to function effectively at rack scale.

It’s also worth mentioning that NVIDIA’s partner ecosystem played a significant role here; for instance, Nebius also contributed with results that showcased the impressive capabilities of the Vera Rubin NVL72.

Given the evolving landscape of AI agents that require multi-step reasoning, planning, and actions, the context by which inference performance is judged is shifting. Notably, during preview tests, Vera Rubin NVL72 achieved a remarkable 30x better performance than the GB300 NVL72 in benchmarks designed for these agentic workloads, such as the SemiAnalysis AgentX. The incoming MLPerf Endpoints benchmark is anticipated to standardize metrics for agentic inference tasks, providing a more accurate assessment beyond traditional throughput measurements.

NVIDIA GB300 NVL72: A Model of Scaling Efficiency

Efficiency in scaling, or the efficacy with which additional GPUs enhance throughput, is a pivotal indicator of AI infrastructure productivity. NVIDIA capitalizes on its high-bandwidth, low-latency scale-up interconnects within each rack, supported by robust networking between racks and efficient request orchestration across nodes.

The DeepSeek-R1 submission illustrates this scaling performance, transitioning from a single GB300 NVL72 rack (72 GPUs) to four racks (288 GPUs) while achieving a remarkable 99% efficiency in the offline scenario. Throughput gains mirrored the hardware growth closely, contrasting with scenarios where infrastructure investment might not yield proportional returns due to inefficiencies in scaling.

The GB300 NVL72 also exhibited impressive rack-scale efficiency on the WAN 2.2 text-to-video benchmark, achieving 0.65 720p videos per second with a processing time of 5.7 seconds per video — yielding 9x higher throughput and a 7.5x reduction in latency over single-node performance.

Continuous Software Improvements Drive Performance Gains

NVIDIA's commitment to software development yields continual performance enhancement and feature upgrades. In the recent MLPerf Inference submission, GB300 NVL72 saw improvements on the Qwen3-VL benchmark by up to 1.6x over the previous v6.0 results, driven by factors such as reduced KV cache precision, kernel fusings, and optimized resource management with vLLM and NVIDIA Dynamo.

Notably, even following the v6.1 submission deadline, post-submission results for other models such as GPT-OSS-120B and DLRMv3 suggested ongoing performance improvements; these results are pending verification by MLCommons.

With the introduction of NVIDIA's Edge-Agentic benchmark, which incorporates configurations like Jetson AGX Thor using TensorRT Edge-LLM, the company continues to demonstrate its prowess in AI inference across various scales. The broad participation of 19 partners, including major players like ASUS, Dell Technologies, and Oracle Cloud Infrastructure, in demonstrating outstanding performances further solidifies NVIDIA’s position in the market.

From compact edge devices to expansive AI infrastructures, NVIDIA remains dedicated to advancing performance throughout its technology stack, iterating on architectural platforms annually while continuously refining software and collaborating with an extensive ecosystem to ensure delivery at scale.

For more insights, learn about the NVIDIA Vera Rubin platform.