Amazon introduces open-source tool for AI chips

Amazon unveiled an open-source tool called TorchNeuron on Monday, enabling developers to run PyTorch code on its AI chips called Trainium without rewriting their code. The tool was presented at the PyTorch Conference North America on October 19 and is set to be released under the Apache 2.0 license on October 20.

TorchNeuron targets a key advantage of NVIDIA

TorchNeuron aims to challenge NVIDIA's dominance in the AI chip market by making Amazon's Trainium chips more accessible. According to Amazon, PyTorch is the language used by about 85% of AI researchers and developers, but adapting PyTorch code to different chips often requires significant rewriting and testing, a process that has historically favored NVIDIA.

TorchNeuron uses standard PyTorch interfaces

TorchNeuron uses a standard socket that PyTorch reserves for new hardware, allowing Trainium chips to be recognized as any other chip. This means developers can use their existing tools to check code, track errors, and distribute work across multiple machines without modifications. Amazon demonstrated this by running the same training program on both a conventional chip and Trainium, with only a setting change to indicate the chip used.

Testing ensures reliability for users

Before releasing new versions of TorchNeuron, Amazon tests them on hundreds of thousands of models to prevent unexpected issues. This approach addresses the real cost of AI development, which for many companies, especially startups, is not just the price of hardware but the time spent adapting software to new chips.

Efficiency benefits large companies

Large companies benefit from using the same chips for both training AI models and inference, the phase where AI responds to users. During off-peak hours, these chips can train new models instead of running idle, and switching between chips without penalty could lower costs and speed up projects.

TorchNeuron gains traction in various sectors

Even before its official release, TorchNeuron has found users in sectors such as video, robotics, autonomous vehicles, speech processing, finance, and large language models like ChatGPT. The company Cartesia, which develops AI capable of interacting with the environment like a human, has run its software chain on Trainium with minimal code changes.

Health research benefits from TorchNeuron

In the health sector, the Lieber Institute for Brain Development is using TorchNeuron to test AI on Trainium that could accelerate molecular dynamics simulations. These simulations, which model the movement of molecules at the atomic level, are computationally intensive and could be made more efficient with Trainium's capabilities.

Amazon supports the broader PyTorch ecosystem

Amazon is not only focusing on its own tool but also supporting the broader ecosystem around PyTorch. Hugging Face, a major platform for sharing AI models, now directly supports Trainium without requiring specific Amazon code. TorchNeuron also works with TorchTitan, a PyTorch tool for distributed training across multiple machines.

Amazon contributes to open-source inference tools

For inference, Amazon has made significant contributions to vLLM, a popular open-source software that accelerates AI responses. The company has published its "kernels," highly optimized programs that run close to the chip, and is working to remove chip-specific code from PyTorch to make it more neutral and accessible to all developers.

TorchNeuron expands with new chip support

TorchNeuron will be published under the Apache 2.0 license on October 20, with support for Trainium2 and Trainium3. This follows a decade of investment in custom chip development, beginning with the acquisition of Annapurna Labs in 2015. Amazon's silicon business now generates over $25 billion annually in revenue.