How to Evaluate AI Tools Without the Hype

The demo always works. That is the first thing to internalise. Every AI product is demonstrated on a curated example, with a cooperative dataset, by someone who knows exactly which prompt produces the good output. The demo is not evidence. It is theatre with a high production value.

What follows is a buying process designed to survive that. It assumes you are an operator with a budget, a real problem, and no interest in becoming an ML engineer.

Start with the workflow, not the tool

The most expensive mistake in AI procurement is buying capability and then hunting for a use case. It produces shelfware with a monthly invoice.

Write the workflow down first, in the boring form: who does this today, how often, how long it takes, what it costs, and what happens when it goes wrong. If you cannot fill in those five fields, you are not ready to evaluate anything. A useful starting point is the workflow catalogue in practical AI workflows for marketing teams.

Only once the workflow is documented does the question “which tool” become answerable.

The seven questions

Ask these in every vendor conversation. Ask them in this order.

1. What is the failure mode, and how will I know it happened?

Every system fails. The question is whether it fails loudly or quietly. A tool that returns an error when it cannot do something is manageable. A tool that returns a confident, plausible, wrong answer is a liability, because your team will act on it.

Ask the vendor to show you a case where the product gets it wrong. If they cannot produce one, they either do not know their own product or they are not being straight with you. Both are disqualifying.

2. What happens to my data?

Where is it processed, where is it stored, is it retained, is it used for training, who at the vendor can see it, and what happens on termination. Get answers in the contract, not in the sales call. For anything touching customer data, this question often ends the evaluation on its own.

3. What does this replace?

If the answer is “nothing, it is additive”, the payback case is weak. Real leverage shows up as removed hours, removed tools, or removed cycle time. Additive tools have a way of becoming permanent line items with no owner.

4. What is the total cost at 10x usage?

AI pricing is frequently usage-metered, and pilots run at a fraction of production volume. Model the cost at the volume you would actually run in year two. Teams that skip this discover the economics only after the workflow is embedded and switching is painful.

Also ask what happens when the underlying model changes. Does your price change? Does your output change?

5. How much setup does this need before it is useful?

Almost every enterprise AI product requires configuration, integration, and context loading before it beats a general model with a decent prompt. Estimate that work honestly in days of your team’s time and add it to the cost side. A product needing eight weeks of internal effort to beat a free alternative has a much higher bar to clear.

6. Can we leave?

Where does our configuration live? Can we export our prompts, our evaluation sets, our fine-tuned artefacts, our history? A tool you cannot exit is not a tool; it is a dependency. Ask for the export format before you sign, not when you are unhappy.

7. What is the thin baseline?

The most important question and the one vendors like least. What does the current best alternative produce, at what quality, for what cost? Frequently the honest baseline is “a well-prompted general-purpose model plus a spreadsheet”, and a surprising number of specialised products struggle to beat it by enough to justify the premium.

Always run the baseline in your pilot. A pilot without a control is a demo you paid for.

The pilot design that actually tells you something

Four weeks, not six months. Structure it like a test, because it is one.

Week 0: define success in writing. Pick one primary metric (hours saved, cycle time, error rate, output volume at a fixed quality bar) and a threshold you would consider a pass. Write it down and circulate it. Post-hoc success criteria always produce a pass.

Week 1: build the evaluation set. Twenty to fifty real examples from your own work, with known good answers or with expert-rated quality. This is the single most valuable artefact you will produce, and it outlives the pilot. When you evaluate the next tool in two years, you will still have it.

Weeks 2 to 3: run tool and baseline in parallel. Same tasks, same people where possible, blind-rated if you can manage it. Track the setup time as a cost, not as a footnote.

Week 4: decide, and write the decision down. Include what you would need to see to reverse it.

Two rules keep pilots honest. First, do not let the vendor run the pilot. Support is fine; operation is not. Second, pilot with the people who would use it daily, not with the enthusiast who volunteered.

Six hype markers worth pricing in

Treat these as signals to slow down, not automatic rejections.

• Benchmarks without your data. Public benchmark performance says close to nothing about performance on your documents, your categories, your customers.
• Accuracy quoted without a baseline. “94 percent accurate” is meaningless unless you know what the current process achieves and what the base rate is.
• Agentic autonomy as the headline. Autonomy raises the cost of a quiet failure. It should be sold with controls, logging, and rollback, not as a feature by itself.
• Roadmap answers to current questions. “That is coming in Q3” means it does not exist. Evaluate what ships today.
• No named production references in your sector. Logos on a website are not references. Ask to speak with a customer who runs it in production at your scale.
• Pricing that requires a call. Not disqualifying, but it usually means the price is a function of your perceived budget rather than the value delivered.

The build-versus-buy line

A rough heuristic that holds up.

Buy when the capability is a commodity (transcription, translation, extraction, generic drafting), when compliance and support matter more than customisation, or when the workflow is not a source of competitive advantage.

Build (which usually means assembling: a model API, your data, a thin interface) when the differentiator is your proprietary data or process, when the vendor premium exceeds roughly three times the assembled cost at your volume, or when no product fits your actual workflow and you would be bending your process to match the tool.

Wait when the workflow is not yet stable, when volume is too low to matter, or when the category is moving fast enough that a decision now locks you in at the worst moment. Waiting is a legitimate decision, and choosing it deliberately is far better than drifting into it. That is the same discipline as deciding what not to do.

The one-page decision record

Whatever you conclude, capture it in a page: the workflow, the baseline, the metric, the result, the cost at projected volume, the decision, and the reversal trigger. Six months later, when someone asks why you chose this, you will have an answer that is not a memory.

Good AI procurement is not about predicting which model wins. It is about buying against a documented workflow, with a real baseline, at a price that survives scale, from a vendor you can leave.

The full build-versus-buy framework, the ten operating domains where AI actually helps, and the workflow patterns behind them are in AI for Operators.