NVIDIA NVLink Fusion Enhances AI Infrastructure with NVHBM Custom High-Bandwidth Memory

The demand for higher performance in AI infrastructure is accelerating as organizations increasingly adopt AI agents and trillion-parameter models. This evolution highlights the necessity for a comprehensive integration of compute, memory, storage, networking, and software into a cohesive system, where the interplay between these elements significantly affects performance.

To address the needs of hyperscalers and AI innovators, NVIDIA has unveiled an upgrade to NVIDIA NVLink Fusion with the introduction of NVIDIA NVHBM. This next-generation high-bandwidth memory technology is designed to enhance the performance and efficiency of XPUs (compute units) within AI systems. NVIDIA has partnered with leading memory providers to validate and offer NVHBM, enabling greater accessibility for NVLink Fusion customers.

Traditional high-bandwidth memory (HBM) architectures typically rely on placing the memory controller directly on the XPU die. While this design optimizes certain functions, it consumes valuable silicon area that could be used for additional compute resources. In contrast, NVHBM adopts a distinct architectural strategy. By integrating NVIDIA's custom memory controller into the base die of the 3D HBM stack—as opposed to the XPU die—NVHBM can deliver significant performance benefits. Specifically, it achieves up to 30% greater memory bandwidth alongside a 15% reduction in power consumption, while freeing up to 25% additional silicon area on the XPU compute die when compared to the earlier HBM4E standard.

NVIDIA aims to establish a standardized implementation of NVHBM, which will be available from multiple memory suppliers. This standardization is expected to minimize the engineering effort needed to integrate and qualify memory solutions across various vendors. Consequently, NVLink Fusion customers will have a streamlined path for accelerating the development and commercialization of custom AI chips.

Amazon’s Annapurna Labs has been identified as the first partner to integrate NVHBM within its broader collaboration with NVIDIA. This initiative underscores the alignment of both companies in enhancing infrastructure capabilities for demanding AI workloads.

Partnerships Enhancing AI Infrastructure

The ongoing collaboration between AWS and NVIDIA, particularly through Annapurna Labs, focuses on harnessing NVHBM technology alongside the NVLink scale-up architecture. This partnership aims to elevate AI performance and efficiency. Key to this is AWS’s prior commitment to support NVLink Fusion, which will utilize Annapurna's upcoming Trainium chips, starting with Trainium4. This integration is notable as it will enable Amazon’s chips and NVIDIA GPUs to interact cohesively within a common rack-scale architecture.

As Nafea Bshara, vice president of Annapurna Labs at Amazon, noted, “NVHBM represents a new architectural approach to advancing high-bandwidth memory performance and efficiency. We look forward to this technology collaboration to benefit future AWS infrastructure designs.” Such statements reflect the enthusiasm surrounding NVHBM’s potential to transform infrastructure capabilities for AI applications.

Unified Systems Approach with NVLink Fusion

NVLink Fusion not only facilitates seamless connectivity of custom XPUs and CPUs but also integrates them into NVIDIA's overarching rack-scale platform. This allows various partners access to NVIDIA NVLink chiplets, NVLink-C2C, as well as NVLink switches and NVIDIA MGX systems. The inclusion of a diverse ecosystem of CPU partners, ASIC designers, and technology providers creates a collaborative framework that benefits all stakeholders involved.

Each iteration of NVIDIA's rack-scale system architecture embraces NVLink Fusion, allowing hyperscalers and AI-native companies to concentrate their engineering resources on developing XPUs. This strategy enables them to leverage an established technology stack for scale-up and scale-out applications, ultimately providing a quicker, lower-risk pathway for the deployment of semi-custom AI infrastructures.

As the demands for AI-centric applications continue to grow, the significance of high-performance memory solutions, like NVHBM, will become increasingly evident. By merging advanced memory architecture with strategic partnerships, NVIDIA is positioning its NVLink Fusion platform as a pivotal player in the next generation of AI infrastructure development.

To find out more about NVLink and NVLink Fusion, visit NVLink and NVLink Fusion.