NVIDIA’s Memory Codesign Targets Higher AI Inference Throughput per Dollar

NVIDIA’s case for improving AI inference economics is that adding high-bandwidth memory (HBM) alone won’t solve the memory bottleneck. Its full-stack approach combines GPU, CPU, networking, storage and software design to keep data close to the compute when needed and move less active data to larger, lower-cost tiers. NVIDIA reports that this strategy helped raise agentic throughput per dollar by up to 30x from GB300 NVL72 to Vera Rubin NVL72, while HBM capacity per GPU increased by about 3%.

That figure is the company’s headline comparison, not a measure of every workload or deployment. It comes from SemiAnalysis AgentX, and the reported result depends on serving performance at specified response rates. The practical idea behind it is broader: reduce the memory each request consumes, then manage the remaining data across a hierarchy instead of keeping everything in the GPU’s most expensive memory.

Why agentic inference strains HBM

During inference, HBM holds model parameters and the key-value (KV) cache, which stores context so a model can continue a request without processing the same material again. Both demands are growing. Frontier models can have trillions of parameters, while agentic systems may preserve context across repeated model and tool calls. NVIDIA says some such trajectories process up to 3 million tokens. Retrieval-augmented generation (RAG) adds relevant enterprise information to prompts, creating further context to manage. NVIDIA’s RAG explainer describes the approach.

More HBM can hold more cache and support more concurrent users, but each user also consumes compute and memory bandwidth. The data has to reach the processors quickly enough to keep them busy. Once bandwidth becomes the constraint, extra capacity may sit underused while adding cost and power demand. NVIDIA calls this limit the memory wall.

The response is to treat memory as a hierarchy. HBM is fast and scarce; CPU memory, local solid-state drives and shared storage offer progressively more capacity at lower cost per bit, with different access speeds. The system’s job is to keep active data near the GPU and retrieve less active data before it is needed again.

Reducing the memory footprint

NVIDIA’s first set of techniques reduces the space occupied by model weights. Its NVFP4 format uses less space than FP8, which NVIDIA describes as the lowest precision available on its Hopper platform. The company points to NVFP4 checkpoints for FLUX.2 [klein] from Black Forest Labs, Inkling from Thinking Machines Lab, Beam 1.0 from Reflection, Meta Muse Spark, Step 5 Preview from StepFun and NVIDIA Nemotron 3.5 Super VL. NVIDIA presents these releases as evidence of model builders adopting four-bit precision for inference.

Weights can also be distributed across as many as 72 GPUs in an NVL72 system. This distribution means each GPU stores a smaller share, leaving more of its HBM for KV cache and concurrent requests. The benefit depends on the system being able to coordinate work across those GPUs.

Cache can also be compressed. NVIDIA says applying NVFP4 to KV cache reduces its footprint compared with FP8, allowing more context to remain in HBM. For paused requests, the Vera CPU provides a larger LPDDR memory tier connected to Rubin GPUs over NVLink-C2C. NVIDIA says faster tool execution can also shorten the time that cache remains idle during tool calls.

Moving inactive context and RAG data

When context is inactive, NVIDIA Dynamo can move it to NVIDIA CMX, a shared context storage tier intended to cost less per bit than HBM. The system returns the context before reuse. NVIDIA says CMX is shared across servers, unlike local SSDs, and avoids some long-term data protection overhead because this context is short-lived. Named storage providers include Cloudian, DDN, Dell Technologies, Everpure (formerly Pure Storage), HPE, IBM, NetApp, VAST and WEKA.

Dynamo’s KV Cache Reuse feature coordinates cache across GPU memory, CPU memory and storage, so context computed once can be reused elsewhere in the system. For RAG, NVIDIA says Nemotron Retriever models create smaller embeddings with higher retrieval accuracy, while its cuVS library supports building and serving the search index across CPU and storage. The knowledge base stays outside HBM; only retrieved passages enter the model’s context.

Reported throughput gains and their limits

On the SemiAnalysis AgentX DeepSeek V4 Pro workload, NVIDIA reports that GB300 NVL72 delivers up to 10x higher throughput per dollar than the H200 platform at 100 tokens per second per user, with HBM capacity per GPU roughly doubling. At 125 tokens per second per user, Vera Rubin NVL72 delivers up to 30x higher throughput per dollar than GB300 NVL72, while HBM capacity per GPU rises about 3%.

Those comparisons make the company’s point: capacity growth alone doesn’t explain the reported performance gains. The result depends on precision, cache placement, data movement and system throughput working together. For operators, the useful measure is how many tokens a system can serve within its power budget while meeting response-time needs. Whether these gains carry over to a particular deployment will depend on its models, traffic and service targets.

For more detail, see NVIDIA’s video explainer and technical deep dive. The key operational question is whether tiered memory and lower precision can keep more of the available HBM serving active work at the response rates users expect.