Robot Pilot vs Production: Acceptance Criteria That Prevent False Success

A robot pilot can look successful and still be unfit for production. A guided demonstration may complete a route, move the expected payload and impress stakeholders while quietly depending on low traffic, expert supervision, manual resets or excluded edge cases. Production acceptance must answer a harder question: can the defined service operate repeatedly, under representative conditions, with controlled exceptions and accountable owners?

The remedy is to write the acceptance gate before the pilot starts. Define the operating scope, evidence, mandatory controls, failure rules and decision authority. Then the pilot becomes a structured test of production readiness rather than a search for encouraging anecdotes.

Separate demonstration, pilot and production acceptance

A demonstration proves that a capability can be shown. A pilot tests a proposed service in a bounded environment. Production acceptance grants permission to operate that service within a named scope. These are different claims and should have different evidence.

Start with the production statement you need to support: for example, “The system may perform these payload movements on these routes, during these shifts, with these interfaces and this support model.” Every acceptance criterion should test part of that statement. Routes, loads, operating windows, traffic conditions, network dependencies, doors, lifts, people, maintenance coverage and prohibited scenarios all belong in the scope.

Build a gate from requirements, risks and operations

NASA’s acceptance-criteria guidance emphasizes establishing criteria before acceptance testing and tracing them to requirements. The same discipline is useful for robotics programs even though NASA guidance is not a commercial robot certification. Create a traceable matrix linking each requirement or material risk to a test method, evidence owner, pass condition and disposition.

Use three types of criteria. Performance criteria cover service quality and timing. Control criteria cover safety, security, access, change control and required procedures. Operating-model criteria cover training, support, incident ownership, spares, monitoring and recovery. A strong overall completion rate must not compensate for a failed critical control.

Test repeatability, not a best run

One clean run establishes possibility, not stability. Define the variation the future service will experience: different shifts, route directions, payload classes, traffic periods, operators, battery states and interface conditions. Run enough representative missions to expose recurring intervention patterns and long-tail delays. Report exclusions and cancellations rather than silently removing them from the denominator.

Record the complete evidence chain: request, dispatch, movement, handoff, exception and close. Pair system logs with observation where logs cannot explain human help or environmental conditions. If the evidence window is too narrow or data are missing, record “insufficient evidence” instead of converting uncertainty into a pass.

Make exception recovery a first-class acceptance test

Production service is defined by what happens when normal flow breaks. Test approved scenarios such as a blocked route, unavailable destination, failed handoff, communications loss, low battery, localization recovery and manual service substitution. Use a controlled method; do not manufacture unsafe conditions.

For every scenario, define detection, safe state, notification, human owner, time expectation, evidence capture and restart authority. The test passes only when the combined robot-and-operations system recovers as designed. A technician rescuing the system through an undocumented workaround is a finding, not a successful recovery.

Use a decision table that prevents averaging away risk

Keep the production claim bounded

Acceptance applies only to the tested configuration and operating envelope. A new payload, building, lift interface, software release or support model may require impact review and partial retesting. Maintain a configuration baseline and define which changes invalidate prior evidence.

The final review should include open risks, deviations, unresolved data-quality issues and conditions of acceptance. Avoid “conditional pass” as a vague compromise: name each condition, accountable owner, deadline and consequence if it remains open.

Use Warpify’s robot integration approach to connect requirements, site readiness and acceptance evidence. When the future operating scope is clear, plan your production acceptance gate.