Token as a key cost measure

The concept of "token," a unit of measure for costs related to large language models (LLMs), is gaining importance in the software industry. Tokens reflect the processing cost of AI tasks, making them a critical factor in infrastructure and financial operations (FinOps) as costs rise rapidly.

Uber’s AI budget usage

Uber used up its annual AI budget in just four months, with tools like Claude Code or Cursor being used by 95% of its 5,000 engineers every month. This rapid consumption has prompted proposals for cost control measures, such as a proposed cap of $1,500 per tool, per month, per employee.

Productivity versus cost challenge

Andrew Macdonald, Uber’s operations director, acknowledged the difficulty of distinguishing between AI usage statistics and actual productivity gains. He noted that while AI may increase engineer costs, it may also boost their productivity, complicating hiring decisions and AI cost management.

Token as a budget blind spot

The Cloud introduced similar challenges, where the exact cost of AI usage was unknown until after the fact. GreenSI described tokens as a budget blind spot, a problem many chief information officers (CIOs) now face. Tokens should be a clear counter, visible on invoices and in management consoles, but misreading them can lead to extremes like panic or overconsumption.

Tokenmaxxing and performance metrics

Amazon introduced counters for AI agents developed by employees, aiming to increase productivity by delegating small tasks. However, this led to employees creating agents solely to climb internal rankings, even if the agents were not useful. This behavior gave rise to "tokenmaxxing," where AI usage became a performance metric.

Shift to token-based pricing

Providers of LLM models announced the end of flat rates by summer 2025, switching to token-based payment once limits are exceeded. This shift caused significant cost increases, with Meta removing rankings and introducing budgets per division. Anthropic spent hundreds of millions of dollars monthly, and Instagram’s head noted the challenge of creating value from token consumption.

Focus on customer results

Microsoft’s Executive Vice President, Jay Parikh, emphasized that token maximization was not the goal. Instead, the focus should be on results for customers and the company. This marked the beginning of a more rational approach to AI usage, shifting from cost-driven metrics to outcome-based goals.

Billing unit changes

Publishers are changing their billing units, which has significant implications for CIOs. GitHub now counts Copilot usage in "AI Credits" per token, with credits included in licenses and budgets managed by cost centers or users. Salesforce and Intercom have also adjusted their pricing models, moving away from flat rates to per-action or per-request charges.

Token counter challenges

The difficulty in reading the token counter stems from three main issues: the price of tokens has decreased, but bills have increased; the measurement of usage can become the objective itself, leading to the Goodhart law; and cost measurement may overlook quality and value. Innovation budgets for language models have fallen from 25% to 7% in a year.

Inference costs rising

Gartner predicts that by 2026, inference—the use of models—will account for 55% of cloud infrastructure costs dedicated to AI. Nvidia’s focus on inference cards aligns with this prediction. However, managing the recurring costs of AI usage lacks the urgency seen during the proof of concept phase.

Balancing cost and quality

The case of Klarna shows the limits of AI agents when cost becomes the dominant factor. The CEO admitted that cost had been "too predominant," highlighting the need for a balanced approach that considers both cost and quality in AI implementation.

Token governance factors

The governance of tokens involves three factors: volume, intensity, and price. Volume relates to business needs, intensity to the CIO, and price to procurement. The cost per business result is a shared indicator that allows for a more nuanced understanding of AI costs and their impact.

Managing token flow

Token governance requires identifying three "pumps" through which tokens flow: direct calls to models, agents with their own identities, and credits integrated into SaaS. Each pump has its own counter and requires specific governance. The challenge is in measuring the right indicators, including cost per use case and business result.

Need for use case registers

A register of use cases is crucial, with each use case having a business owner, a DSI application manager, and a buyer. Baselines should be measured before testing, and written stop criteria should be established from the start. Gartner predicts that more than 40% of agent AI projects will be abandoned by 2027, emphasizing the need for rational governance.

AI governance at a crossroads

The governance of AI usage is now a critical issue, with tokens moving from being a hidden metric to a central concern. The challenge lies in managing a complex equation with three stakeholders, which is never simple. The industry is at a crossroads, needing to balance AI benefits with its costs and governance complexities.