The cost of AI's 'compute' is coming into focus, and it's a lot

Artificial intelligence is expensive. We know this. The chips are expensive, the data centers are expensive, the water to cool systems is expensive. Putting all hardware and software together to create this computer program that seems to speak like a human and can do virtually anything infinitely faster than a human is extremely expensive. That is why the AI buildout is costing trillions.

Previously, this wasn't entirely clear. The tech companies building and selling these products were so eager to get them into customers' hands that the companies were eating a large portion of the costs, selling their products for less than the cost of producing them.

But that's ending. Tech companies are changing the way they charge for AI, and the customers, including banks, are starting to get a look at the real costs. That is prompting many of them to start looking for cheaper alternatives, our Nathan Place reports. Most of these companies charged flat per-person seat licenses; if you've heard of "tokenmaxxing," it's a result of this kind of charging practice. You're being charged a flat rate no matter how much or how little you use it, so why not use it, like, a lot. This apparently has become too much for the AI companies, so Anthropic, Open AI, Microsoft and others have changed their pricing model. Now banks and other companies are being charged per "token," basically by how much data they use.

The companies buying AI licenses now face a dilemma. Some have blown through a year's worth of their AI budget in months. They are discovering that the true cost is far more than the heavily subsidized cost they were paying before. And that's before you even consider whether or not their AI usage is producing any productivity gains. So they're looking for ways to cut corners. PNC is building its own AI factory, taking much of its usage in-house.

Part of the problem for companies trying to buy AI systems is that the cost of "compute," as the cool kids call it, is very opaque, but there are efforts to bring transparency to AI costs, and there is a certain logic to it. If AI is going to become as ubiquitous as its proselytizers proclaim, then the "compute" is almost like a commodity that can and should be clearly priced like any other commodity.

There are a few people working on this now. A startup called Ornn has started compiling indices that put visible priceds on tokens. The firm says its indices are built on data derived from executed transactions for GPU computing. Another startup, called IFX, is building derivatives products on top of the Ornn data, giving market types a way to play the compute market the same way traders play any other commodity market.

This whole dynamic, though, is a risk for the AI companies. Spending on the AI buildout is now in the $2 trillion range and rising, according to a report from a newsletter writer named Azeem Azhar. That's in the range of estimates from a number of other sources. While a lot of the early money was supplied by the hyperscalers themselves, more of it increasingly being financed, from banks and private credit. And what that means is that the pressure ratchets higher for the AI companies to show a profit. They can't just subsidize the costs anymore.

Azhar estimates it will cost roughly $8 billion to run 1 gigawatt's worth of AI capacity annually. To realize a 25% return on investments will require a "compute" price somewhere in the range of $1.05-$2.10 per token. Good news! According to Ornn, the price for an H100 GPU as of Sunday was $2.45.

But companies are already figuring out ways to get AI cheaper. Nathan's story details several strategies banks are using to lower their AI bills, including looking at open-source models, using older models that still perform usefully, and, like I already said, building their own. For the companies that are borrowing hundreds of billions, possibly even trillions, to build these products, that's a risk.

It's also, incidentally, a risk for the creditors that lend that money out.