Measure the baseline before the AI pilot starts

Three months into an AI pilot for drafting customer replies, the steering group asks the obvious question: is it saving time? The team believes it is. The people using it say it helps, mostly. The vendor’s report shows thousands of drafts generated. Nobody can say how long a reply took before the pilot, how many were reworked, or how many complaints came back — because nobody measured any of it before the pilot started.

The pilot is extended for another quarter, which is what happens to pilots that cannot be judged.

Why this keeps happening

Measuring the current state feels like delay. The budget is approved, the tool is ready, people are enthusiastic, and two weeks of counting how the work is done today looks like bureaucracy. So the pilot starts, and the baseline is lost the moment it does: once people use the tool, the old way of working no longer exists to be measured.

The cost arrives later, when a decision about scaling has to be made on impressions.

How unmeasured pilots go wrong

Activity is reported instead of effect. Prompts, drafts, active users. These show that the tool is used, not that the work is better or cheaper. A tool can be heavily used and change nothing, or make the work slower.

Time saved is counted; time added is not. Drafting gets faster. Reviewing, correcting and handling the cases where the draft was wrong get slower. Unless both are measured, the net effect is a guess, and it is usually guessed generously.

Enthusiasm stands in for evidence. The people who volunteered for the pilot are the people who like the tool. Their opinion is real, and it is not representative of the people who will be asked to use it next.

Quality drifts without anyone noticing. If nobody recorded the error rate, the complaint rate or the rework before, a small decline afterwards looks like nothing. Some of the cost of AI arrives as slightly worse output rather than as a visible failure.

The comparison moves during the pilot. Seasonal volume, a new product, a change in staffing. Without a baseline from a comparable period, any of these can make the pilot look better or worse than it is.

What to measure, and how little is enough

Not a measurement programme. Two to four numbers, chosen before the pilot starts, measured over a couple of weeks of normal work.

Time per unit of work, end to end. Not the drafting step — the whole thing from arrival to done, including review and correction. Sample it if timing everything is impractical; twenty cases measured honestly beat a thousand estimated.

A quality signal the business already trusts. Rework, returns, complaints, corrections by a reviewer, whatever exists. If nothing exists, define one before starting and accept that it is imperfect.

How much of the AI output gets changed. Once the pilot runs, track how heavily drafts are edited. Heavy editing is the clearest sign that the tool is costing time rather than saving it, a point the article on AI agents makes too.

The cases the tool cannot handle. Count them, and count the time they take. This is where a pilot that looks good on average quietly loses.

Then agree, in writing and before any results are in, which result means scale, which means adjust, and which means stop. A stop condition written early is the only kind anyone ever uses. It fits the short increments described in how to buy consulting so that it has to work: every increment ends in a decision, and a decision needs something to compare against.

Why this belongs inside the pilot

A working pilot is one of the things VUCALE does, and measuring the starting point is its first week rather than an optional preface. A pilot that cannot be compared with anything produces an opinion. A pilot with a baseline produces a decision — including, sometimes, the decision that the tool is not worth it for this process, which is a perfectly good result for six weeks of work.

A test you can run today

Pick the AI tool or pilot your company is most excited about right now. Ask the person responsible two questions: how long did this work take before, and how will we know in three months whether it got better?

If the first answer is a number from a measurement, you are in good shape. If it is an estimate, there is still time to measure, as long as the tool is not yet part of daily work. If the pilot has already started, find a team or a period where it is not used and measure that instead. A rough baseline taken now is worth more than a precise one nobody took.