PyTorch model compatibility with new hardware

A PyTorch model originally designed for Nvidia GPUs can now run on Amazon or Google chips with minimal changes to the code or configuration, according to both companies. This compatibility is achieved through new tools like TorchNeuron and TorchTPU, which allow models to execute on different hardware platforms with limited rewrites.

Deepgram’s port of the Nova-3 model

Deepgram, the first customer to publicly detail the process, successfully ran its transcription model Nova-3 on Amazon’s Trainium2 chips. This involved significant rework, including rewriting calculation cores with AWS engineers, developing a new convolution layer from scratch, and building a service engine in Rust.

Rewriting custom kernels for new hardware

Both Amazon and Google require custom kernels written in CUDA or Triton to be rewritten in the language specific to their respective chips. These kernels are essential for optimizing performance on each platform, and neither company has yet provided documentation or tools for automatically translating these kernels.

TorchNeuron and its requirements

TorchNeuron, currently in public beta, requires PyTorch 2.14.0 and is scheduled to support version 2.15 by October 28. It replaces older bridges like PyTorch/XLA and uses the PrivateUse1 interface reserved for third-party hardware in PyTorch.

TorchTPU and its integration with Transformers

Google’s TorchTPU, co-developed with Meta’s PyTorch team, replaces PyTorch/XLA for TPU support. It uses the same PyTorch/XLA compilation pipeline, including OpenXLA and StableHLO, but skips Inductor. Google recommends migrating existing workloads to TorchTPU.

Support for PyTorch models on different platforms

Hugging Face reports that 89% of the 100 most used text generation models are natively supported with Transformers, compared to 46% with the older Optimum Neuron library. This suggests that newer tools like TorchNeuron and TorchTPU offer better compatibility with current models.

Limitations and ongoing development

Both TorchNeuron and TorchTPU have known limitations. TorchNeuron only supports specific Python versions and Amazon’s Trn2 and Trn3 instances, while TorchTPU supports multiple TPU generations but lacks clear version requirements. Some operations still require recompilation, and certain features remain in experimental stages.

Performance claims and benchmarks

Amazon claims that its system can run 100,000 Hugging Face models with high accuracy, though no success rate is published. Google mentions performance improvements for training models like Qwen 3 8B, but details are limited. Deepgram reports a 60% to 70% improvement in audio processing efficiency, but the exact benchmarks are not fully transparent.

AMD and Intel’s x86 optimization project

AMD and Intel announced plans to consolidate their PyTorch optimizations for x86 processors into the oneDNN library, following recommendations from the x86 Ecosystem Advisory Group. This move aims to provide a unified path for x86 acceleration in PyTorch, with updates expected to include new instruction sets for matrix multiplication.