Is AI More Hype Than Reality? Why Organisations Are Now Measuring AI Productivity — and Counting Every Token

Every few years the technology industry falls in love with an idea so completely that questioning it feels almost rude. Cloud had its moment. Blockchain had a louder one. Today it's artificial intelligence — and specifically generative AI — that sits at the centre of nearly every board deck, product roadmap and conference keynote.

So it's fair to ask the uncomfortable question: is AI creating more hype than reality?

My honest answer is both — and the more interesting story isn't whether AI works, but what happened when organisations started actually measuring it. Somewhere between the demo and the invoice, the conversation changed from "What can AI do?" to "What is AI doing for us, and what is it costing?"

The hype was real — and so were the expectations

The early wave of generative AI was genuinely impressive. A model that could draft an email, summarise a contract, write a working function or explain a stack trace felt like a step change, and in many ways it was.

The problem was the leap from impressive demo to transformed business. Vendors promised autonomous agents replacing whole workflows. Leadership teams set targets like "30% productivity gains" before anyone had defined what productivity meant. Budgets were approved on the strength of a prototype that worked beautifully on ten hand-picked examples.

Gartner captured the mood shift neatly. In its 2025 Hype Cycle for AI, generative AI slid past the Peak of Inflated Expectations and into the Trough of Disillusionment. That isn't a verdict that the technology failed — every durable technology passes through the trough — but it marks the point where excitement gives way to scrutiny.

Where reality fell short

A few pieces of research did more than anything else to puncture the easy optimism:

• The pilot-to-production gap. Gartner predicted in 2024 that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025, citing poor data quality, weak risk controls, escalating costs and unclear business value. In 2025 it went further, predicting that over 40% of agentic AI projects will be cancelled by the end of 2027.
• The "GenAI Divide". MIT's NANDA initiative published a widely discussed 2025 report suggesting that the vast majority of enterprise generative AI pilots — around 95% by its measure — were producing no measurable impact on profit and loss. The methodology has been debated, but the headline struck a nerve because it matched what many teams were quietly seeing.
• Feeling faster isn't being faster. In a 2025 randomised controlled trial, the research group METR studied experienced open-source developers working on their own mature codebases. With AI tools allowed, they took about 19% longer to complete tasks — yet they believed the tools had sped them up by around 20%.

That last result is the one I think about most. It doesn't prove AI coding tools are useless; plenty of studies show gains in other settings, especially for less familiar code or more junior engineers. What it proves is that perceived productivity is a terrible proxy for actual productivity. If the people using the tools can't reliably tell whether they're faster, a company certainly can't rely on anecdotes and satisfaction surveys.

Why organisations started watching AI productivity more carefully

In the first wave, AI spend was treated like R&D — experimental, a bit of a bet, hard to question. That's no longer true, for a few reasons.

1. AI moved from the innovation budget to the operating budget

Once AI features ship to customers or get rolled out to thousands of employees, they become a recurring cost line. Recurring costs get reviewed. CFOs who happily funded a pilot now want to see the same rigour they'd apply to any other operating expense.

2. Seat licences don't equal usage

Many organisations bought AI assistants for everyone. A few months later, the usage dashboards told a familiar story: a group of enthusiastic power users, a large middle that tried it a few times, and a long tail that never came back. Paying for a seat nobody uses is the easiest waste to spot — and it triggered deeper questions about the value of the seats that are used.

3. The "last mile" is where the effort hides

AI output often looks 90% done. The final 10% — checking facts, fixing edge cases, reviewing generated code, correcting tone — is where the human time goes. If you only measure how fast the first draft appears, you'll overestimate the benefit every time. Teams began tracking rework, review time and defect rates, not just generation speed.

4. Risk and quality became measurable costs

Hallucinated answers in a customer chat, insecure generated code, or confidential data pasted into the wrong tool all carry real costs. Monitoring quality and compliance became part of the same exercise as monitoring productivity.

The token problem: cheaper per unit, more expensive overall

This is the part that surprises people. The price of AI, measured per token, has fallen dramatically. Analysis from venture firm Andreessen Horowitz described it as "LLMflation" — the cost of reaching a given level of model capability falling by roughly 10x per year.

And yet many organisations are watching their total AI bills go up. How?

Because the way we use models changed faster than the price fell:

• Reasoning models think out loud. Models that deliberate before answering can generate many times more tokens per request than earlier chat models — and you pay for that thinking.
• Agents loop. An agent that plans, calls tools, reads results and retries can make dozens of model calls to finish one task. A single "do this for me" can quietly become hundreds of thousands of tokens.
• Context windows got huge — and so did prompts. It's tempting to stuff entire documents, codebases or chat histories into every request. Long context is powerful, but every token in it is billed, every time.
• Usage scales with success. When a feature works, people use it more. That's the goal — but a cost model that looked fine at 1,000 users can look very different at 100,000.

It's a textbook case of the Jevons paradox: when something gets cheaper to use, we tend to use far more of it, and total consumption rises.

A quick illustration

The numbers below are hypothetical, but the shape is typical:

	Simple chat assistant	Agentic workflow
Model calls per task	1	25
Tokens per call (in + out)	3,000	12,000
Tokens per task	3,000	300,000
Tasks per month	50,000	50,000
Tokens per month	150 million	15 billion

Even if the agent's model is ten times cheaper per token, the workflow still uses a hundred times more tokens — so the bill is ten times higher. Whether that's a bargain or a disaster depends entirely on what each completed task is worth. That's the key point: token cost only means something next to the value of the outcome.

What mature AI monitoring actually looks like

The organisations getting this right have stopped asking "How much are we spending on AI?" and started asking "What does each unit of value cost us?" In practice, that means tracking a handful of things together:

What to measure	Why it matters
Cost per outcome — per resolved ticket, merged PR, processed document	Connects token spend directly to business value
Active usage — weekly active users vs. licences bought	Exposes shelfware and adoption gaps
Acceptance and rework rates	Shows how much AI output is actually kept, not just generated
End-to-end cycle time	Captures the review and fixing time that demos hide
Quality signals — defects, escalations, customer satisfaction	Makes sure speed isn't bought with errors
Tokens per task and per feature	Catches runaway prompts and agent loops early

A few practices have become standard in teams that run AI at scale:

1. Tag every request. Attribute each model call to a team, feature and use case, the same way cloud costs are tagged. You can't optimise what you can't attribute.
2. Set budgets and alerts. Treat token spend like any other cloud resource — with per-feature budgets, anomaly alerts and hard limits on agent loops.
3. Route by difficulty. Not every request needs the biggest model. Sending simple tasks to smaller, cheaper models and saving frontier models for the hard ones is often the single biggest cost lever.
4. Cache and trim. Prompt caching, shorter system prompts and retrieving only the relevant context can cut token usage substantially without hurting quality.
5. Run controlled comparisons. Measure the same workflow with and without AI, on real work, over weeks — not a one-off demo.

In short, AI is getting the FinOps treatment that cloud computing got a decade ago.

So — hype or reality?

Here's where I land.

The hype was the belief that AI would deliver transformation on its own — that buying licences and wiring up a model would automatically produce a leaner, faster organisation.

The reality is more modest and, frankly, more useful. AI delivers strong returns in specific, well-defined workflows: drafting and summarising, search over internal knowledge, customer-support triage, code assistance on the right kinds of tasks, and document processing. It delivers those returns when the workflow is redesigned around it, when quality is checked, and when cost is tracked per outcome.

The trough of disillusionment isn't the end of the story. It's the part where the tourists leave and the builders stay. The organisations that come out ahead won't be the ones that spent the most on AI — they'll be the ones that learned to measure it honestly.

If your team is working through this right now, start small: pick one workflow, define what "better" means in numbers, measure the cost per outcome, and let the data — not the demo — decide what scales next.

What's your experience been? Are you seeing real productivity gains from AI, or mostly rising bills? I'd love to hear about it in the comments.