A new benchmark developed by researchers at Columbia University and the University of Cambridge is designed to test how well machine learning (ML) models can predict the thermal and mechanical properties of materials. The benchmark, detailed in a study published in Nature Communications, uses a "physics-aware" method to evaluate whether ML models accurately describe the atomic-level behavior of materials, especially how atoms vibrate, when making predictions about larger properties like thermal conductivity. The researchers, including Michele Simoncelli from Columbia and Balázs Póta from Cambridge, pointed out that while ML models can generate predictions that look accurate, they might be based on flawed reasoning. The benchmark has already been used by various research teams and companies, including Meta, Microsoft, and startups like Radical AI and Orbital Materials, to assess their models. Understanding the quantum behavior of solids usually involves solving the Schrödinger equation, a complex mathematical problem that is too slow for practical use with real materials. ML models, known as machine-learning interatomic potentials, aim to speed this up by learning from large datasets that include atomic positions, energies, forces, and stresses derived from quantum-mechanical calculations. These models can be up to 1,000 times faster than traditional methods, but they might still have errors in predicting how atoms respond to forces, which are key to understanding how materials behave under heat or pressure. The researchers tested multiple ML models against traditional quantum-mechanical calculations for over 100 different crystalline materials. Some models correctly predicted both atomic vibrations and macroscopic thermal conductivity, while others made significant errors in predicting atomic vibrations but still managed to predict thermal conductivity accurately due to microscopic errors canceling each other out. Through targeted training for specific materials, the team achieved predictions that were within a few percent of known results, including for lithium bromide. The benchmark is now part of Matbench Discovery, an online platform that ranks ML models based on their ability to predict crystal stability, structure, and thermal conductivity. It has highlighted significant performance differences between models that seemed similar in predicting crystal stability, revealing the weaknesses of models that lack strong physical constraints. Thermal properties are essential in many fields, from microchip cooling and battery design to quantum computing and aerospace engineering. By offering a way to early on evaluate the usefulness of ML models, this benchmark is expected to help advance technologies that rely on material properties, ensuring that predictions are not only fast but also physically accurate.