Maximizing Frontier Models on AMD with Inference Alpha by DigitalOcean

DigitalOcean Enhances AI Inference Performance with AMD GPUs

DigitalOcean has announced significant advancements in artificial intelligence (AI) infrastructure, focusing on optimizing inference performance for frontier Large Language Models (LLMs) using AMD GPUs. The company aims to provide high-performance solutions that not only enhance speed but also improve cost-efficiency, addressing the complex challenges associated with deploying such models in production environments.

The Challenge of Inference Performance

Inference performance is a multifaceted issue that goes beyond merely utilizing powerful hardware. DigitalOcean identifies several key factors that contribute to optimal performance, including model architecture, runtime execution, memory systems, scheduling, and decoding strategies. This intricate interplay means that achieving peak output speed requires a comprehensive understanding of the entire system rather than relying solely on raw hardware capabilities.

DigitalOcean’s approach emphasizes the concept of “performance alpha,” which refers to the potential gains derived from specialized inference engineering. By customizing the software stack instead of adhering to standard configurations, the company asserts that it can match or even exceed the performance of more expensive hardware setups while maintaining accuracy.

Collaborative Optimizations with Wafer

To validate its theories on performance optimization, DigitalOcean collaborated with Wafer, a company specializing in AI infrastructure solutions. Together, they implemented various optimizations on specific frontier models running on AMD GPUs. By leveraging Wafer’s Agent technology to identify inefficiencies and apply targeted fixes, they achieved substantial improvements in inference performance.

• Kimi 2.5 (High-Speed Single Stream)

  • A stock configuration using 8x MI350X/MI355x hardware achieved a baseline of 22.5 tokens per second (tok/s) on a standard workload. After deep kernel optimization and a tailored inference framework, this figure surged to 255.2 tok/s—an impressive 11.33x speedup without sacrificing accuracy.

• DeepSeek V3.2 (Full-Stack Scaling)

  • The stock frameworks recorded an output speed of 38.5 tok/s for single requests; however, DigitalOcean’s optimized stack boosted this to 200.8 tok/s. At a concurrency level of 64, there was a remarkable 7.32x improvement in per-request output speed and an increase in aggregate throughput from 548 tok/s to 2,165 tok/s.

• GLM-5 (Flagship Efficiency)

  • This massive model features 774 billion parameters and was optimized for deployment topology and decoding paths. As a result, a single 8-GPU MI350X node managed a mean throughput of 151.1 tok/s with an inter-token latency of just 17.8 milliseconds.

Transforming Inference Economics

The outcomes of these optimizations signify more than just technical achievements; they represent a transformative shift in the economics surrounding frontier inference tasks. DigitalOcean’s efforts demonstrate that fully optimized AMD infrastructure can deliver elite performance levels while remaining more cost-effective than traditional flagship hardware deployments.

The emphasis is clear: achieving high performance alongside sustainable economics necessitates a deep understanding of the software stack and how it interacts with underlying hardware resources.

Identifying Bottlenecks in Standard Frameworks

DigitalOcean defines “stock” frameworks as unmodified versions of inference engines or standard kernel libraries that are often used as quick-start solutions for deploying models. However, these stock frameworks come with inherent limitations that can hinder performance:

• The Generality Trade-off: Stock kernels are designed for broad compatibility but may not optimize well for the specific needs of frontier architectures.
• Prefill Bias: Standard kernels are calibrated for large prefill batches but can become inefficient when applied to single-stream decoding tasks.
• The Launch Tax: Stock setups often involve multiple kernel launches for operations like all-reduce and residual add, leading to unnecessary overhead.
• Rigid Software Constraints: Many standard libraries impose hard-coded requirements that can create incompatibilities with new frontier models.

A Roadmap for Future Optimizations

This initiative marks only the beginning of DigitalOcean’s exploration into advanced inference engineering techniques. The company plans to release three technical “surgeries” focusing on different frontier models and their specific optimizations:

• Part 1: The Kimi 2.5 Deep-Dive: An analysis of how an 11x speedup was achieved through custom MLA (Multi-Head Latent Attention) and MoE (Mixture of Experts) kernels.
• Part 2: Scaling DeepSeek V3.2: Insights into full-stack serving optimizations and high-concurrency throughput enhancements.
• Part 3: Optimizing GLM-5: A breakdown of deployment strategies involving TP=4 topologies and specialized batched GEMV kernels for maximum efficiency.

What This Means for AI Infrastructure

The advancements made by DigitalOcean in optimizing AI inference performance highlight the importance of tailored approaches over generic solutions in achieving high efficiency at scale. As organizations increasingly rely on LLMs for various applications, understanding these nuances will be crucial for maximizing both performance and cost-effectiveness in AI deployments.

For more information, read the original report here.