Maximizing Frontier Models on AMD with Inference Alpha from DigitalOcean

DigitalOcean Advances AI Infrastructure with AMD GPUs

DigitalOcean has announced significant advancements in its infrastructure aimed at enhancing the performance of artificial intelligence (AI) applications, particularly those utilizing large language models (LLMs). By leveraging high-performance AMD GPUs, the company aims to optimize inference performance, which is crucial for deploying these models effectively in production environments. This initiative underscores DigitalOcean’s commitment to providing cost-effective solutions that rival more expensive hardware options.

Understanding Inference Performance Challenges

Inference performance presents a complex systems-level challenge that goes beyond merely having powerful hardware. Factors such as model architecture, runtime execution, memory systems, scheduling, and decoding strategies all play critical roles in determining overall performance. DigitalOcean emphasizes the importance of specialized inference engineering to unlock what it terms “performance alpha,” which refers to achieving significantly better performance through tailored software optimizations rather than relying solely on standard configurations.

The company believes that by customizing the software stack for specific workloads, it can deliver competitive performance levels while maintaining cost efficiency. This approach addresses the non-obvious hurdles often encountered in the current software ecosystem, enabling better economics for high-performance AMD infrastructure compared to traditional flagship deployments.

Collaboration with Wafer for Enhanced Performance

To validate its “Performance Alpha” theory, DigitalOcean collaborated with Wafer, a company specializing in AI optimization. Together, they focused on enhancing specific frontier models running on AMD GPUs through various optimizations. Utilizing Wafer’s Agent technology, they identified inefficiencies and implemented fixes that resulted in substantial performance improvements across several key benchmarks.

• Kimi 2.5 (High-Speed Single Stream)

  • A stock configuration using 8x MI350X/MI355x hardware achieved a baseline of 22.5 tokens per second (tok/s) on a standard workload. Through extensive kernel optimization and a customized inference framework, this was increased to 255.2 tok/s—an impressive 11.33x speedup without sacrificing accuracy.

• DeepSeek V3.2 (Full-Stack Scaling)

  • The optimized stack improved single-request output speed from 38.5 tok/s to 200.8 tok/s. At a concurrency level of 64, per-request output speed saw a 7.32x improvement, with aggregate throughput jumping from 548 tok/s to an astounding 2,165 tok/s.

• GLM-5 (Flagship Efficiency)

  • This massive model boasts 774 billion parameters. By optimizing deployment topology and specializing the decode path, a single 8-GPU MI350X node achieved a mean throughput of 151.1 tok/s and an inter-token latency of just 17.8 milliseconds.

Redefining Inference Economics

The results achieved through these optimizations signify a fundamental shift in the economics of frontier inference. DigitalOcean’s work illustrates that fully optimized AMD infrastructure can deliver elite performance levels while remaining more cost-effective than traditional flagship hardware setups. The emphasis on systems-level thinking highlights that achieving both high performance and sustainable economics necessitates a deep understanding of the underlying software stack and how it interacts with hardware capabilities.

Identifying Bottlenecks in Standard Frameworks

DigitalOcean’s research has identified several bottlenecks inherent in standard frameworks used for AI inference:

• The Generality Trade-off: Stock kernels are designed for broad compatibility but often lack optimization for specific model dimensions.
• Prefill Bias: Standard kernels are calibrated for large prefill batches but can lead to inefficiencies during single-stream decoding.
• The Launch Tax: Stock setups often incur overhead from dispatching operations as separate kernels, resulting in unnecessary latency.
• Rigid Software Constraints: Many libraries contain hard-coded assertions that create compatibility issues with new frontier models.

The Future Roadmap for Inference Engineering

This initiative marks just the beginning of DigitalOcean’s exploration into inference engineering optimizations. The company plans to release three technical deep dives focusing on different frontier models and their respective optimizations:

• Kimi 2.5 Deep-Dive: An exploration of achieving an 11x speedup through custom kernel engineering.
• Scaling DeepSeek V3.2: Insights into full-stack serving optimizations and high-concurrency throughput gains.
• Optimizing GLM-5: A breakdown of deployment topologies and fine-tuning techniques for maximum efficiency.

What This Means

The advancements made by DigitalOcean highlight an important trend in AI infrastructure: optimizing software stacks can yield substantial performance improvements without necessitating expensive hardware upgrades. As organizations increasingly rely on LLMs for various applications, understanding these optimizations will be crucial for maintaining competitive advantages while managing costs effectively.

For more information, read the original report here.