NVIDIA Blackwell Ultra Breaks Inference Records in MLPerf Introduction

As we delve into the world of artificial intelligence, it’s evident that large language models (LLMs) are transforming the landscape with their increasing complexity and capabilities. These models, developed by leading tech innovators, now boast hundreds of billions of parameters, showcasing their growing intelligence and ability to perform complex reasoning tasks. This evolution in AI demands a leap in computational power to efficiently manage these sophisticated models.

The focus on enhancing computational performance is evident in the technology stack that supports these models. It encompasses state-of-the-art chips, advanced systems, and cutting-edge software, all of which are essential for achieving peak performance. Moreover, a thriving developer ecosystem is vital for continuously building and refining this technology stack.

Introduction to MLPerf Inference v5.1

The MLPerf Inference v5.1 is the latest iteration of an industry-standard benchmark designed to evaluate AI inference performance. Benchmarks are crucial as they offer a structured way to measure and compare the performance of different AI models and systems. Held biannually, MLPerf benchmarks are regularly updated to include new models and scenarios, reflecting the rapid advancements in AI technology. This latest round of benchmarks introduces several noteworthy models:

1. DeepSeek-R1: Developed by DeepSeek, this model is a mixture-of-experts (MoE) with a staggering 671 billion parameters. In server scenarios, it targets a time-to-first-token (TTFT) of 2 seconds with a throughput of 12.5 tokens per second per user (TPS/user). This means that 99% of the tokens meet or surpass this speed, emphasizing the model’s efficiency.
2. Llama 3.1 405B: Part of the Llama 3.1 series, this model introduces a new interactive scenario with a faster TPS/user threshold of 12.5 and a reduced TTFT of 4.5 seconds, presenting a significant improvement over previous server scenarios.
3. Llama 3.1 8B: This 8-billion parameter model, also from the Llama 3.1 series, is designed for various scenarios, including offline, server, and interactive settings. It replaces the GPT-J benchmark used in earlier rounds.
4. Whisper: This speech recognition model has gained immense popularity, with nearly 5 million downloads in a single month on platforms like HuggingFace. It replaces the RNN-T model that was part of previous MLPerf Inference benchmarks.

   NVIDIA’s Groundbreaking Submissions

   In this round, NVIDIA made significant contributions by submitting results using their newly introduced Blackwell Ultra architecture. This architecture, announced in March, builds upon the successes of the previous Blackwell architecture. It features several enhancements, including a 1.5x increase in peak NVFP4 AI compute, double the attention-layer compute, and 1.5x greater HBM3e capacity. These improvements allow the Blackwell Ultra architecture to deliver up to 1.4x higher performance per GPU compared to its predecessor, the GB200 NVL72 submission.

   Innovations in AI Performance

   NVIDIA’s innovative use of the NVFP4 acceleration across all DeepSeek-R1 and Llama model submissions showcases their commitment to pushing the boundaries of AI performance. NVFP4 is a low-precision inference technique developed by NVIDIA, which enables efficient and accurate AI computations. By quantizing model weights to NVFP4, NVIDIA achieved reduced model sizes and enhanced throughput, all while maintaining high accuracy levels.

   Another key innovation is the use of FP8 key-value caches. By storing the key-value cache in FP8 precision, NVIDIA significantly reduced memory usage, enabling higher performance. The company also introduced new parallelism techniques to maximize the efficiency of multi-GPU execution, ensuring balanced workloads and minimizing bottlenecks.

   Advanced Serving Techniques

   NVIDIA’s submissions also showcased the power of disaggregated serving, particularly in the Llama 3.1 405B interactive benchmark scenario. By decoupling the context and generation phases of inference across separate GPUs or nodes, NVIDIA optimized each phase independently. This approach allowed for different parallelism techniques and flexible GPU allocation, resulting in improved overall system efficiency.

   The NVIDIA Dynamo inference framework further supports disaggregated serving, offering features like SLA-based autoscaling and real-time observability metrics. These capabilities enhance the scalability and reliability of AI inference deployments.

   Key Takeaways

   NVIDIA’s contributions to the MLPerf Inference v5.1 benchmarks highlight their leadership in AI inference performance across a wide range of models and scenarios. The introduction of the Blackwell Ultra GPU architecture and innovative serving techniques demonstrate significant advancements in AI technology. These developments promise to accelerate the capabilities of AI models, paving the way for more sophisticated and efficient AI applications.

   For those interested in exploring the technical details and reproducing the results, the MLPerf Inference v5.1 GitHub repository provides valuable resources. Additionally, NVIDIA’s unveiling of the Rubin CPX processor, designed to enhance long context processing, marks another step forward in AI performance and efficiency.

   In conclusion, as AI continues to evolve, the need for advanced computational capabilities becomes increasingly apparent. With companies like NVIDIA leading the charge, the future of AI looks promising, offering new possibilities and opportunities for innovation across industries.

For more Information, Refer to this article.