In the rapidly evolving field of artificial intelligence (AI), the complexity and collaborative nature of AI models are increasing at an unprecedented pace. To meet the demands of this new era, AI inference—the process of using AI models to make predictions or decisions—must evolve accordingly. This evolution requires scaling across entire clusters to efficiently serve millions of users simultaneously and provide swift responses.
Much like the advancements seen in large-scale AI training, Kubernetes has emerged as the industry standard for managing containerized applications. It is now well-positioned to handle the multi-node inference necessary to support sophisticated AI models. This article explores NVIDIA’s innovative approach to enhancing AI inference performance through its Dynamo platform, which works hand-in-hand with Kubernetes.
### Optimizing AI Performance with Disaggregated Inference
AI models that fit within the capacity of a single GPU or server often run multiple identical copies in parallel across numerous nodes. This strategy is employed to achieve high throughput. A recent study by Russ Fellows, a principal analyst at Signal65, demonstrated a groundbreaking aggregate throughput of 1.1 million tokens per second using 72 NVIDIA Blackwell Ultra GPUs. This achievement underscores the potential of such parallelization.
However, as AI models expand to accommodate real-time services for many users, or when managing resource-intensive tasks with long input sequences, a technique known as disaggregated serving offers a pathway to further enhance performance and efficiency. Disaggregated serving involves splitting the AI model’s workload into distinct phases: processing the input prompt (prefill) and generating the output (decode). Traditionally, both tasks were executed on the same GPUs, leading to inefficiencies and bottlenecks.
Disaggregated serving intelligently assigns each task to independently optimized GPUs. This ensures that every workload component benefits from the most suitable optimization techniques, thereby maximizing overall performance. For contemporary AI reasoning models like DeepSeek-R1, disaggregated serving proves indispensable.
NVIDIA’s Dynamo platform seamlessly integrates these multi-node inference optimization features, including disaggregated serving, enabling efficient production-scale deployment across GPU clusters. The result is evident in its substantial value delivery.
For instance, Baseten leveraged NVIDIA Dynamo to accelerate inference serving for long-context code generation by a factor of two and increased throughput by 1.6 times, all without incurring additional hardware costs. Such software-driven performance enhancements allow AI providers to significantly cut the costs of developing intelligence.
Moreover, recent benchmarks from SemiAnalysis InferenceMAX highlighted that disaggregated serving with NVIDIA Dynamo on NVIDIA GB200 NVL72 systems offers the lowest cost per million tokens for mixture-of-experts reasoning models like DeepSeek-R1, among the platforms tested.
### Scaling Disaggregated Inference in the Cloud
As disaggregated serving scales across numerous nodes for enterprise-scale AI deployments, Kubernetes plays a crucial role as the orchestration layer. NVIDIA Dynamo’s integration into managed Kubernetes services offered by major cloud providers enables customers to scale multi-node inference across NVIDIA Blackwell systems, such as GB200 and GB300 NVL72, with the performance, flexibility, and reliability necessary for enterprise AI deployments.
Several cloud providers are already leveraging these capabilities:
– Amazon Web Services (AWS) is using NVIDIA Dynamo alongside Amazon EKS to accelerate generative AI inference for its customers.
– Google Cloud provides a NVIDIA Dynamo recipe to optimize large language model (LLM) inference at the enterprise scale on its AI Hypercomputer.
– Oracle Cloud Infrastructure (OCI) supports multi-node LLM inferencing using OCI Superclusters and NVIDIA Dynamo.
The drive to enable large-scale, multi-node inference extends beyond hyperscalers. For example, Nebius is designing its cloud infrastructure to handle inference workloads at scale, built on NVIDIA’s accelerated computing infrastructure and collaborating with NVIDIA Dynamo as an ecosystem partner.
### Simplifying Inference on Kubernetes with NVIDIA Grove in NVIDIA Dynamo
Disaggregated AI inference requires the coordination of specialized components, such as prefill, decode, and routing, each with distinct requirements. The challenge for Kubernetes has evolved beyond running parallel copies of a model. It now involves orchestrating these diverse components into one cohesive, high-performance system.
NVIDIA Grove, an application programming interface now available within NVIDIA Dynamo, allows users to define a single, high-level specification that outlines their entire inference system. For example, a user can specify their needs: “I need three GPU nodes for prefill and six GPU nodes for decode, and all nodes for a single model replica must be placed on the same high-speed interconnect for the fastest response.”
Grove automatically manages the intricate coordination: scaling related components together while maintaining correct ratios and dependencies, initiating them in the proper sequence, and placing them strategically across the cluster for efficient communication. For a deeper understanding of NVIDIA Grove, a technical deep dive is available.
As AI inference becomes more distributed, the combination of Kubernetes, NVIDIA Dynamo, and NVIDIA Grove simplifies how developers build and scale intelligent applications. These technologies come together to make cluster-scale AI accessible and production-ready.
For those keen to explore these advancements further, NVIDIA will be present at KubeCon, running through Thursday, November 13, in Atlanta. This event offers a unique opportunity to delve into how these technologies are revolutionizing AI inference at scale.
By implementing these cutting-edge technologies, organizations can significantly boost their AI inference capabilities, offering faster, more efficient services to millions of users worldwide. As NVIDIA continues to innovate, the future of AI looks even more promising, with unprecedented opportunities for businesses and developers alike.
For more Information, Refer to this article.
