NVIDIA Advances AI Infrastructure Efficiency with Vera Rubin and DSX Platforms at AI Infra Summit

At the AI Infra Summit held at the Santa Clara Convention Center, Ian Buck, NVIDIA's vice president for hyperscale and high-performance computing, presented key developments in AI factory efficiency. With attendance surging to over 8,000 from just 3,500 last year, the summit has evolved into a significant venue for discussing infrastructure technology.

Collaborative Innovations

Buck highlighted various partnerships aimed at enhancing the capabilities of NVIDIA’s platforms:

• Amazon’s Annapurna Labs is collaborating on developing custom high-bandwidth memory technology, termed NVHBM.
• d-Matrix is integrating NVIDIA's NVLink Fusion to synergize NVIDIA Vera CPUs with d-Matrix Raptor XPUs, facilitating ultra-low-latency inference at scale.

Highlights of New AI Factory Deployments

Several groundbreaking results were revealed during the summit:

• Lambda showcased a 23% enhancement in performance per watt utilizing NVIDIA’s DSX MaxLPS.
• Pinterest is leveraging NVIDIA's Blackwell platform along with Dynamo inference software to incorporate conversational AI into visual discovery.

The increasing adoption of agentic AI workloads necessitates enhanced performance, efficiency, and scalability from AI infrastructure. NVIDIA has responded by offering a comprehensive AI factory platform, including Vera Rubin systems, Dynamo inference software, NeMo libraries, and advanced networking solutions like NVLink, Spectrum-X Ethernet, and BlueField DPUs.

“We’re building infrastructure that is reliable and sustainable for a decade, forming an asset to tackle the world's computing challenges,” Buck stated, reinforcing the importance of the innovations brought forth.

Shifting Metrics in AI Infrastructure

As the industry evolves, the focus of AI infrastructure is transitioning from peak performance to a new metric: validated agentic tokens per megawatt. NVIDIA’s DSX MaxLPS aims to enhance performance by up to 1.4 times more tokens per megawatt through company-wide power optimization, while the NVLink technology enables substantial accelerated computing integration.

This shift reflects a growing emphasis on generating tokens more efficiently while maximizing the value of each megawatt of power consumed.

MLPerf Inference v6.1 Benchmarking

NVIDIA also reported impressive results in the latest MLPerf Inference v6.1 benchmarks, underscoring the importance of system performance, infrastructure scalability, and software optimization in determining AI inference economics. The Vera Rubin NVL72 system demonstrated a remarkable throughput increase of up to 3.7 times over the previous generation GB300 NVL72, reinforcing its superior capabilities in next-generation AI infrastructure.

The GB300 NVL72 also achieved a notable 99% scaling efficiency across a 288-GPU configuration spanning four racks, indicating an almost linear growth in throughput with an increase in resources.

An essential aspect of this achievement was software innovation, with NVIDIA’s benchmarking submission showing a performance boost of 1.6 times compared to v6.0, primarily driven by software enhancements.

AI Factory Flexible-Load Program

Emerald AI’s partnership with NVIDIA has birthed a flexible-load interconnection program aimed at enhancing grid flexibility. Through this initiative, Emerald AI demonstrated a system that responded automatically to demand signals from Silicon Valley Power, adjusting its operational load while ensuring consistent AI performance.

Leveraging NVIDIA's DSX Flex technology, Emerald AI's Conductor software dynamically manages energy consumption in accordance with real-time electricity grid conditions. This feature allows the AI factory to responsibly balance energy demand, pausing lower-priority jobs while retaining processing power for critical tasks. This adaptability not only aids in energy efficiency but also opens avenues for AI factories to serve as flexible resources for grid relief.

Lambda’s Token Throughput Maximization

At the AI Infra Summit, Lambda presented results validating NVIDIA’s DSX MaxLPS technology. The DSX MaxLPS tool continuously adjusts power allocation across GPUs and racks, optimizing for various workload demands inherent in AI training and inference tasks. Lambda reported that it could sustain 19 nodes within the power budget typically set for 16 nodes, leading to a 24% increase in cluster token throughput from 4 million to 5 million tokens per second, along with a 23% improvement in performance per watt.

Consequently, this advancement hints that future deployments of NVIDIA's Vera Rubin NVL72 could unlock 40% more GPU capacity within the same megawatt budget.

NVIDIA Vera Rubin's Power Efficiency

In leveraging its full-stack AI factory design, NVIDIA enhances power management through its Vera Rubin NVL72 systems, where every watt of consumption directly impacts performance. The DSX MaxLPS technology allows for dynamic power distribution across systems, achieving enhanced performance metrics without the need for additional power infrastructure.

This approach has yielded results such as:

• A 40% increase in GPU utilization within current site power restraints.
• A 35% boost in token throughput without requiring new power infrastructure.

NVIDIA's Intelligent Power Smoothing software further allows racks to capitalize on short-term power spikes, leading to more productive computations by minimizing stranded power capacity.

Enhanced AI Capability with Groq 3 LPX

The combination of NVIDIA's Vera Rubin architecture and Groq 3 LPX significantly enhances throughput, particularly for demanding agentic AI tasks characterized by reasoning and tool use. The systems deliver up to 35 times higher token throughput per megawatt with two-trillion-parameter models operating on long context lengths. For instance, Groq 3 LPX achieved a impressive 2,529 output tokens per second for specific workloads, expanding the processing capabilities for complex AI tasks.

SemiAnalysis AgentX Benchmarking

The AI factory performance of Vera Rubin NVL72 has also been assessed through SemiAnalysis's AgentX, focusing on real-world agentic coding scenarios. The NVL72 outperformed its GB300 counterpart by 30 times in throughput per megawatt and noted a 45-fold reduction in costs per million generated tokens.

These results highlight the performance gains drive innovations within AI factory economics. The architecture includes well-engineered interconnectivity and precision technology, further improving operational efficiencies.

Startups Testing NVIDIA Vera CPUs

Startups have begun employing NVIDIA Vera CPUs in various demanding applications, reporting outstanding performance metrics. For example:

• Perplexity documented a 1.9 times improvement in startup times for its SPACE platform.
• ClickHouse observed the Vera CPU achieving the fastest measurements for its analytical database benchmarks.
• Redpanda noted significant advantages in throughput and latency compared to other CPUs.

The Role of NVIDIA NVLink 6

As AI factories rapidly scale, NVIDIA's NVLink 6 addresses challenges related to reliability and performance. This architecture offers multilayer resiliency, enhancing the robustness of training and inference workloads while managing transient errors effectively.

The features include advanced error correction, credit-based flow control, and dynamic routing to minimize the impact of faults on overall performance. Collectively, these technologies aim to sustain optimal productivity as infrastructure requirements evolve.

Overall, NVIDIA’s advancements at the AI Infra Summit encapsulate a significant leap in AI infrastructure efficiency, positioning the company as a key player in optimizing power utilization and driving performance in next-generation AI workloads.