Open-weight AI models gain traction

Open-weight AI models, which allow users to download and run AI locally on their own devices, are becoming a popular alternative to cloud-based AI services. These models are seen as potentially more cost-effective, with IBM CEO Arvind Krishna suggesting they can reach about 90% of the performance of top closed models at a much lower cost per token.

Cost efficiency claims lack detail

Krishna's comments, reported by The Register, suggest that open-weight models can cost 10 to 25% less per token than cloud-based alternatives. However, IBM has not provided specific details on how these cost comparisons were made, leaving some uncertainty about the exact methodology used.

Open-weight models are not always open-source

While open-weight models can be run locally without relying on the servers of their creators, they are not always open-source. The training data and methods used to develop these models can remain confidential, even if the model's parameters are made available for download.

More companies are offering open-weight models

The availability of open-weight models has grown in recent months, with companies like IBM, Alibaba, DeepSeek, Mistral, and even OpenAI offering models with accessible weights. This expansion suggests a growing interest in providing AI capabilities that are more flexible and adaptable to different needs.

Running AI locally has trade-offs

For companies, running AI locally offers the benefit of using internal documents without sending them to external providers. However, it also requires significant technical expertise, infrastructure, and resources. Costs for hardware, electricity, and maintenance, along with the need for skilled personnel, must be carefully evaluated.

Big companies are reevaluating AI costs

Krishna noted that large companies are closely examining the profitability of their AI investments, wary of "uncontrollable expenses with no real return." This concern is driving the search for more affordable AI solutions, and IBM, which provides the software and infrastructure for deploying open-weight models, benefits from this trend.

Hybrid approaches may emerge

In the long term, companies may use local models for routine tasks and rely on cloud-based AI for more complex needs. This hybrid model allows for flexibility and cost control. For individuals, tools like LM Studio are enabling the offline use of compatible models, making open-weight AI more accessible.

Quantization improves hardware compatibility

Advancements in quantization techniques are helping to reduce the memory requirements of AI models by decreasing the numerical precision of their parameters. This makes it easier to run models on less powerful hardware, though it can result in a minor loss of quality in performance.

Quantization and hardware needs

Quantization allows models to be executed with less VRAM or RAM, increasing compatibility with a wider range of hardware. However, the most advanced models still require high-performance configurations. The idea of using AI without recurring subscription fees or per-request billing is becoming more realistic with these developments.

Understanding open-weight model licenses

Open-weight models refer to AI models whose trained parameters are published for download and local execution. The license attached to these models is crucial, as it can determine whether they can be used commercially, redistributed, or modified. These models primarily describe access to the final product, not the full training process.

Token costs in AI billing

In generative AI billing, a token is a unit of text used by language models, often the basis for cloud service charges. The cost per token varies depending on the model, the volume of text processed, and whether the model is run locally or through a hosted API. Comparing token costs requires a specific scenario, as general claims like "10 times cheaper" are hard to verify without a defined measurement protocol.