Review an agentic commerce pilot before scaling

An agentic commerce pilot reports an 87.5% success rate. Before using that number to justify expansion, the reviewer needs to know which requests entered the pilot, which attempts were counted and what happened to the unresolved cases.

Agentic commerce pilot evaluation should make the denominator traceable. A clean percentage can conceal excluded customers, missing outcomes or actions that succeeded technically without the required authority.

Fix the cohort and observation window

Use a fictional pilot with 100 distinct, in-scope buyer requests in a defined observation window. Each request has one intended purchase outcome. Repeated technical calls belong to the same request and do not create additional customers in the cohort.

Before the pilot begins, define the outcome evidence, observation deadline and decision criteria. The UK's Test, Learn and Grow guidance supports planning the evidence needed for a decision and avoiding broad conclusions from limited testing. It does not prescribe the synthetic scorecard below. Testing, learning and iterating.

At the window's close, the example contains 20 correct stops before a completion attempt and 80 attempted completions. Among the attempts, 70 have merchant-confirmed authorized orders, six have definite failure evidence and four remain unknown.

Show the counts behind each rate

All figures are illustrative. No pilot result is claimed.

The two completion percentages answer different questions. The first concerns attempted completions; the second concerns the full request cohort. Neither justifies removing the four unknown outcomes from the report. A later update can resolve them, but should preserve what was known at the original reporting time.

Explain the 20 stops separately. In this example, they correctly prevented commitments outside the permitted conditions. In a real pilot, the reviewer would need evidence that each was a correct stop, rather than label every abandoned request a safety success.

Add authority and duplicate-commitment evidence

A merchant-confirmed order belongs in the authorized-success count only when the required permission also holds. Report any authority violation or duplicate commitment separately, even if the customer ultimately received an item.

Use the original request identity to investigate retries. One request that accidentally creates two orders is not two successful completions. Preserve both merchant records and the recovery outcome, and assess the control failure on its own terms.

The UCP checkout test explores how one purchase reaches a confirmed merchant order. The pilot scorecard aggregates outcomes only after that evidence can be tied back to individual requests.

Decide what the evidence actually permits

The scale decision needs more than the headline rate. Examine the reasons for definite failures, the age and recoverability of unknown cases, support work and any unacceptable commitments. Define which findings prevent expansion before interpreting a promising average.

A small successful cohort may support another controlled test in the same scope. It does not establish readiness for a new market, merchant type, order value or permission model. Nor does confirmed purchase performance demonstrate that later refund and support processes work.

The agentic commerce readiness guide covers the broader operating questions. Use this cohort record to make a narrower decision: what was learned from these requests, which uncertainty remains, and which next test would reduce it?

Keep the original cohort visible after the decision. The most useful scorecard allows another reviewer to reconstruct the 87.5%, locate the four unknown requests and see whether the apparent successes were actually authorized.