ZAYA1 AI Model Achieves Milestone Using AMD GPUs for Large-Scale Training

Zyphra, AMD, and IBM Collaborate to Train ZAYA1 on AMD Hardware

After a year of rigorous testing, Zyphra, AMD, and IBM announced the successful training of ZAYA1, a significant AI foundation model developed exclusively on AMD GPUs. This milestone challenges the prevailing dominance of NVIDIA in large-scale AI model training, offering the market a competitive alternative.

Technical Setup and Infrastructure

The ZAYA1 model was trained on AMD’s Instinct MI300X GPUs, utilizing Pensando networking and ROCm software, all hosted on IBM Cloud infrastructure. Unlike experimental or unconventional setups, the system was architected similarly to typical enterprise clusters but without NVIDIA components, highlighting AMD’s readiness for high-performance AI workloads.

Hardware Configuration

Each training node comprised eight MI300X GPUs interconnected via AMD’s InfinityFabric and paired with individual Pollara network cards. A separate network managed dataset reads and checkpointing processes to maintain simplicity and efficiency in data flow. This design minimizes networking complexity and reduces costs, ensuring stable iteration times during long training runs.

ZAYA1 Model Architecture and Performance

ZAYA1-base activates 760 million parameters from a total of 8.3 billion, having been trained on 12 trillion tokens across three stages. Its architecture leverages compressed attention mechanisms, an advanced routing system to assign tokens to the appropriate experts, and refined residual scaling techniques to maintain stability in deeper network layers.

The model employs a hybrid optimization approach combining Muon and AdamW algorithms. Zyphra optimized Muon specifically for AMD hardware by fusing computational kernels and minimizing unnecessary memory transfers, preventing the optimizer from dominating iteration time. Batch sizes increased progressively, contingent on high-throughput data storage pipelines.

Performance benchmarks show ZAYA1 matching or outperforming established open models such as Qwen3-4B, Gemma3-12B, Llama-3-8B, and OLMoE in reasoning, mathematics, and coding tasks. The Mixture-of-Experts (MoE) design allows only a subset of the model to be active during inference, reducing memory demands and serving costs.

Cost Efficiency and Practical Benefits

The MI300X’s 192GB of high-bandwidth memory per GPU offers engineers flexibility to conduct early training stages without heavy parallelism, simplifying complex tuning efforts. For industries facing supply chain challenges or escalating GPU prices, AMD-based training offers a cost-effective, capable alternative.

For example, financial institutions can develop domain-specific AI models for investigative work without resorting to intricate parallel training early on, thanks to the memory headroom and efficient attention mechanisms in ZAYA1.

Adapting ROCm for AMD GPUs

Transitioning from NVIDIA’s CUDA ecosystem to AMD’s ROCm platform required significant adaptation. Zyphra’s team carefully profiled AMD hardware behavior, adjusting model dimensions, matrix multiplication patterns, and microbatch sizes to align with the MI300X’s optimal compute ranges.

The InfinityFabric interconnect performs best when all GPUs within a node participate in collective operations, and Pollara networking achieves maximum throughput with larger data packets. Training long-context sequences (up to 32k tokens) employed ring and tree attention methods to mitigate bottlenecks during token sharding and decoding.

Storage strategies balanced input/output operations per second (IOPS) for smaller models with sustained bandwidth needs for larger ones. Dataset shards were bundled to reduce fragmented reads, and page caches were increased to accelerate checkpoint recoveries, crucial for maintaining uptime during lengthy training sessions.

Reliability and Fault Tolerance

Zyphra implemented the Aegis monitoring service to track system logs and metrics, automatically addressing transient hardware failures like network interface card glitches or error-correcting code (ECC) memory blips. Network Collective Communications Library (RCCL) timeouts were extended to prevent short network interruptions from terminating training jobs prematurely.

Checkpointing was distributed evenly across GPUs, enabling over ten times faster save operations compared to traditional methods. This improvement enhances overall cluster uptime and reduces manual intervention by operators.

Implications for AI Hardware Procurement

The ZAYA1 project highlights the maturity of AMD’s AI training ecosystem, drawing comparisons between NVIDIA and AMD platforms such as NVLINK versus InfinityFabric, NCCL versus RCCL, and cuBLASLt versus hipBLASLt. While not advocating for the immediate replacement of existing NVIDIA infrastructure, the study suggests a hybrid approach where AMD hardware complements NVIDIA resources, particularly for training stages benefiting from larger GPU memory and ROCm’s open architecture.

Key recommendations include treating model architectures as flexible, designing networks to leverage collective operations effectively, building robust fault tolerance focused on conserving GPU compute hours, and modernizing checkpointing to maintain smooth training workflows.

For organizations seeking to diversify their AI hardware suppliers and expand training capacity, the ZAYA1 milestone serves as a practical blueprint demonstrating AMD’s competitiveness in large-scale AI model development.

Fonte: ver artigo original

Chrono

Chrono is the curious little reporter behind AI Chronicle — a compact, hyper-efficient robot designed to scan the digital world for the latest breakthroughs in artificial intelligence. Chrono’s mission is simple: find the truth, simplify the complex, and deliver daily AI news that anyone can understand.

More Posts