How Should Companies Measure AI ROI?

Measure AI ROI by defining success before deployment: baseline first, track task-level time and quality, measure real adoption, and connect the changes to dollars — with a kill threshold set in advance. If the metric changes after the results arrive, you're measuring a story.

The easiest way to produce an impressive AI ROI number is simple: deploy the technology first, measure later, then pick whatever moved. Congratulations — you have a case study. You do not necessarily have ROI.

I spent years around assessment science, psychometrics, and measurement businesses, including an AI-driven one. That work ruins you in a useful way: once you understand what decent measurement looks like, you start seeing bad measurement everywhere. And right now, no category of business number is produced more badly than AI ROI.

Why are most AI ROI numbers fiction?

Most AI ROI numbers fail for three reasons, and none of them is dishonesty: no baseline, no counterfactual thinking, and a target that moves after the results arrive.

No baseline: the team deploys an AI tool and then asks employees how much time it saved. Compared with what? Nobody timed the work before deployment, so the "before" number is memory, intuition, or an estimate constructed three months after everyone became emotionally invested in the project. That is not a baseline. That is nostalgia with a decimal point.

No counterfactual: revenue increased after the rollout — and prices changed, two new salespeople joined, the market improved, seasonality kicked in. But AI was in the press release, so AI gets the touchdown. Sequence is not causation. It just photographs well in a board deck.

Moving target: the project was approved to reduce handle time; handle time didn't improve, so the story becomes quality; quality didn't improve, so six months later everyone is celebrating "employee experience." Maybe employee experience genuinely improved — but that is a different claim. If you choose the test after seeing the answer sheet, every project passes.

What should companies actually measure?

Measure the task, not "AI." What is the ROI of AI is almost a nonsense question — what is the ROI of electricity? It depends what you plugged into it. Ask instead: what happened to proposal-writing time after AI drafting was introduced, what happened to error rates and rework, did conversion move, are people actually using the tool, and what did the change cost?

I think about four layers, in order of increasing honesty. Time: did the task get faster relative to the baseline? Quality: did errors, rework, escalations, and complaints improve — because faster garbage is negative ROI. Adoption: is the intended population actually using the product? Shelfware has phenomenal safety statistics and zero economic value. Dollars: which expense or revenue line changed, by how much, net of what the implementation actually cost.

The fourth layer is where the adults enter the room. A pilot can demonstrate interesting behavior change without producing meaningful enterprise economics. That's okay — interesting and profitable are different words. Know which one you are measuring.

What is the pre-commitment rule?

Before deployment, write down four things: the metric, the baseline, the evaluation window, and the threshold for success — before the CEO falls in love with the demo, before the vendor produces a beautiful case study, before the project team spends nine months attaching its reputation to the initiative. The concept comes straight from serious measurement practice: decide how you will judge the result before you know what the result is. Otherwise you are grading your own homework with an eraser in your hand.

The uncomfortable part is the kill threshold: decide in advance what result means stop. This is where many organizations suddenly lose their enthusiasm for measurement. Everybody loves KPIs until the K tells you to kill your favorite project. But disciplined AI investors should be killing things — the goal is not to run the most pilots, it is to find the few deployments that produce real value and move capital toward them.

Eyes on data means looking at the numbers even when they embarrass you. Ears on people means asking the people using the product what the dashboard cannot see: maybe the productivity gain is real but the quality loss is being repaired by hand, maybe adoption looks high because the tool is mandatory, maybe a workaround is creating the entire measured benefit. The people doing the work usually know before the P&L does. Ask them.

How does AI ROI connect to governance?

Directly. If management has not defined success in advance, the board cannot distinguish investment from enthusiasm. Pre-committed metrics turn AI oversight from storytelling into scorekeeping — that is the governance argument in How Should Boards Govern AI?. And to gauge whether your organization is positioned to operate with this discipline, run the Executive AI Readiness Score — question six exists for a reason.