Rehearse the Failure, Not the Demo: Building Adversarial Pilots for Niche Platforms
Photo: SpaceX Photos, CC0, via Wikimedia Commons
A proof of concept is the last moment in a vendor relationship when your team still has leverage. After the signature, every unanswered question becomes a support ticket, a roadmap promise, or an argument with an account manager. Yet most pilots are structured to produce a single outcome — a green checkmark — and are staffed accordingly: a named solutions engineer, a curated dataset, and a support channel that answers in minutes because someone is watching it.
That is not a test. It is a demonstration with a schedule. The pilot is where you find out what the tool does when nobody is watching, and it only tells you that if you design it to be uncomfortable.
Write the Failure Conditions Before the Success Criteria
Start with a one-page document that inverts the usual order. Before anyone asks what the tool should do well, list the conditions under which you would refuse to trust it. Sample entries: it silently truncates long fields; it cannot reconcile a duplicate record; it requires a human to re-enter data after a timeout; it escalates through a shared inbox. Ten to fifteen items is usually enough to expose whether the vendor has thought about operations or only about adoption.
Circulate that list to the vendor before the kickoff. A vendor that engages substantively with your failure conditions is telling you something useful. A vendor that routes the list to marketing and returns a slide about "robust error handling" is telling you something more useful still.
Load the Data You Would Never Show a Customer
Curated pilot data is clean, deduplicated, English-only, and consistent in its identifiers. Production data has nulls where a required field should be, three naming conventions for the same customer, a column that means two different things depending on which team created the record, and a handful of records with mixed character encodings that arrived from an acquisition in 2019. If the pilot environment has never seen that, you have learned nothing about the environment you will actually run.
Ask for an isolated instance and load a representative extract, not a sanitized one. Watch what the import validation does, how it reports partial failures, and whether it tells you which rows it dropped. Undocumented silent rejection is one of the most expensive behaviors a specialized platform can have, because it surfaces months later as a reconciliation gap nobody can explain.
Test the Escalation Path at the Hour It Will Actually Break
Support during a scheduled pilot window, with an engineer on standby, is not the support you are buying. The path that matters is the one at 2 a.m. on a Saturday during a close or a deployment freeze. You cannot simulate the hour, but you can simulate the path.
Ask for the on-call rotation and the escalation matrix in writing. File a ticket outside business hours with a severity-one framing and time the response. Ask who is authorized to make a configuration change under pressure, and whether that person is on your contract or on a shared queue. Record the answers. If the vendor cannot produce an escalation document during the sales cycle, it will not produce one during an incident.
Recovery Is the Feature You Are Evaluating
Uptime is the number vendors put in the contract. Recovery is the number that determines whether an incident becomes an outage. Ask for a live demonstration of rollback: delete a record in the pilot instance, corrupt an index, revoke a service credential, and ask the vendor to walk through restoration while your engineer watches. Note whether the recovery is self-service, whether it requires a vendor engineer, and how long it takes.
Then ask the question that separates mature platforms from promising ones: what state is unrecoverable? Every system has an answer. The vendors that give you a precise one have usually been through it.
Count Integration Debt as an Acceptance Criterion
A pilot that connects cleanly to one system has demonstrated one connection. Ask how many connectors, webhooks, scheduled jobs, and custom scripts the full deployment requires, and who owns each of them after go-live. Then ask for the deprecation policy: notice period for connector changes, versioning scheme for the API, and whether breaking changes are announced to a mailing list or discovered in a release note.
The count matters more than the demo. A tool that requires eleven fragile integrations to work is a tool that will consume engineering capacity in proportion to that count, regardless of how well the pilot went.
Close With a Decision Memo, Not a Deck
The pilot should end with a written memo from your own team, not a vendor readout. It should state what was tested, what failed, what the vendor committed to in writing, and which failure conditions remain unresolved. Attach the ticket timings and the recovery walkthrough.
That memo becomes the basis for contract terms: the support commitments, the deprecation notice, the rollback language, the data export format. A pilot that produces a slide deck has produced a memory. A pilot that produces a memo has produced a procurement artifact — and a rehearsal your team has already survived once before it mattered.