---
title: "Test the platform on your data, not the vendor's demo"
url: "https://vucale.com/blog/test-the-platform-on-your-data"
description: "Every platform looks right in a demo built for it. A short test on your own messy data, scored against criteria written beforehand, shows what the demo hides."
---

# Test the platform on your data, not the vendor's demo

October 8, 2026·3 min read·Editorial team

A young woman works on a laptop next to a tall stack of paper files while a colleague in an olive shirt leans over her shoulder to look at the screen

Three vendors present in one week. Every demo is polished: data loads instantly, dashboards respond, the AI assistant answers questions in plain language. Every vendor is credible, every reference customer is happy, and at the end of the week the selection committee has three good impressions and no way of choosing between them. The decision is eventually made on price, on relationships, or on which presenter was most convincing.

Two years later the integration with the ERP is still not finished.

## Why a demo cannot answer the question

A demo exists to show the platform at its best, on data prepared for it, by people who know where the rough edges are. That is not dishonest; it is what a demo is for. But your problem is not the vendor’s demo data. It is your data — with its duplicate customers, inconsistent codes, empty fields and the one system nobody wants to touch.

What decides whether a platform works for you is how it behaves on that, and a demo is precisely the setting in which you cannot see it.

## Where selections go wrong

**The criteria are written after the demos.** The committee sees the features, then writes down what it values, and the list unconsciously describes the platform that impressed most. Criteria written afterwards confirm a preference; they do not test it.

**The requirements are a feature checklist.** Three hundred lines, every vendor answers yes to nearly all of them, and the differences that matter — how hard something is to do, not whether it can be done at all — disappear.

**Nobody attempts the hardest task.** Every platform can load a clean file and draw a chart. What separates them is the awkward integration, the messy source, the edge case in your process. It is rarely tested, because it is uncomfortable for vendor and buyer alike.

**The people who will use it are not in the room.** Managers and IT choose. The analysts and operators who will live with the tool see it for the first time after the contract is signed.

**The exit is not part of the evaluation.** How the data comes out, in what format, at what cost. It never appears in a demo, and it becomes the most important question at the first renewal. The [article on reselling](https://vucale.com/blog/a-consultancy-that-resells-cannot-advise-you) explains why the advisor helping you choose may not raise it.

## What to do instead

**Write the scoring criteria first.** Before any vendor presents. Five to eight criteria that reflect your actual problem, with weights, agreed by the people who will decide. Freeze them, and score against them afterwards.

**Build a test pack from your own data.** A realistic extract, anonymised where it has to be, including the ugly parts: the duplicates, the free-text fields, the legacy codes. Add one or two tasks that represent real work, including the one you suspect is hard.

**Run a short, timeboxed test.** Two shortlisted vendors, one to two weeks each, the same test pack, the same tasks. Ask them to do the work with your people watching — better still, have your people do it with the vendor guiding. How long things take, what needs the vendor’s engineers, and what turns out to be “on the roadmap” is the real result.

**Include the people who will use it.** Two or three future users score usability against the same criteria. Their scores carry weight, because adoption depends on them.

**Test the way out.** Ask each vendor to export the test data and everything built on it in an open format. Note how long it takes and what gets lost.

Many vendors will agree to this when asked early and when the effort is bounded. A vendor that will not test on realistic data has told you something useful. And whoever runs the test should have no stake in its result — one of the reasons VUCALE [resells nothing](https://vucale.com/).

## A test you can run before the next demo

Before your next vendor presentation, write down on one page the three tasks the platform must do on your data, how you will know it did them well, and what would rule it out. Send the page to the vendor in advance and ask them to demonstrate exactly those three things.

Then watch what happens to the demo. If it changes substantially, you were about to buy the presentation rather than the platform.

-   vendor selection
-   platforms
-   proof of concept
-   data

## Keep reading

-   [Agent factories will not stay in IT](https://vucale.com/blog/agent-factories-will-not-stay-in-it)
-   [The pilot worked. Scaling is a different project](https://vucale.com/blog/the-pilot-worked-scaling-is-a-different-project)