Choose a Product Metric That Survives an Easy Shortcut

A team builds a workspace for recording decisions. Its activity dashboard shows more messages, comments and reactions every week. During a customer interview, a project lead opens six discussion threads and still cannot find the decision they need.

The team has made activity easier to count. It has not yet shown that the product makes decisions easier to use.

This is an illustrative product review, not a reported company result. It shows how to test a metric before making it a target: describe the user value, identify a measurable signal, then look for ways the signal could improve while the underlying experience gets worse.

Begin with a useful event

For this workspace, one plausible outcome is that an authorized colleague can find the current decision, understand its context and use it for the task at hand. Message volume is several steps away from that outcome. It may be useful diagnostic data, but more messages can also mean repeated questions or an unresolved disagreement.

Amplitude's North Star guidance connects a useful metric to customer value and to inputs a team can influence. It also discusses the limitations of possible metrics. That framework is a starting point for choosing a hypothesis, not proof that a selected proxy causes retention or revenue. Amplitude's metric guidance

A proposed metric for the workspace could be the proportion of sampled decision-seeking tasks in which a participant finds the correct current decision without asking another person to locate it. The phrase is less convenient than “weekly active users,” but it identifies the task and the standard for success.

That metric is still a proxy. Finding a document does not prove the decision was sensible or that the team acted on it correctly. Keep its meaning narrow enough that the evidence can support it.

Specify what enters the denominator

Suppose the team samples 40 genuine decision-seeking tasks over an agreed review window. In 26 tasks, the participant finds the current decision using the product. The observed result is 26 ÷ 40 = 65%. These numbers are invented to demonstrate the calculation.

The percentage is not useful until the team can explain the 40 tasks. Were they selected before outcomes were known? Did they include new colleagues, difficult searches and projects with incomplete documentation? Were abandoned searches retained? If only completed searches enter the denominator, the metric can improve by losing the users who struggled most.

GOV.UK's service-measurement guidance recommends clear definitions, useful segmentation and combining performance data with user research. Its context is public services; the principle applied here is to make the measure interpretable rather than transplant a government metric into a commercial workspace. Using data to improve a service

Record exclusions before the review. A training exercise may be excluded from a production-use measure, for example, but that does not justify excluding every task with an unsuccessful result. Keep counts for missing or unclassifiable observations so data gaps remain visible.

Try to improve the number in the wrong way

A short challenge session can reveal more than another dashboard mock-up. Ask someone unfamiliar with the metric to make it look better with the least useful work possible.

The exercise is a test of the measurement design, not an accusation that the team will cheat. Product choices can create these effects without anyone intending to manipulate a score. A search shortcut might genuinely help some users while obscuring an important permission or version distinction.

Put guardrails beside the headline measure

For the proposed workspace, track wrong-version retrievals, unauthorized exposure and the burden of maintaining decision records alongside the task-success measure. Also examine whether a gain for experienced users comes at the expense of new colleagues.

Guardrails need a response, not just another chart. Decide who investigates a permission failure and whether the affected feature must be restricted while the issue is resolved. Define how a rise in maintenance effort will be assessed. Some additional effort may be acceptable if decisions become more useful; hiding the effort prevents an honest tradeoff.

Do not average away a serious failure by combining it with a high success score. A product can help most searches while exposing one customer's restricted record. The release decision needs to consider that evidence directly.

Define useful user progress

Product identifies the real task and a proposed measurable signal.

Specify the eligible population and observation rule

Can the number improve while the experience worsens?

Try easy shortcuts, omitted failures, wrong versions and permission breaches.

Yes, or the evidence is unclear

Revise the metric or its guardrails. Retest the shortcut before relying on the score.

Challenge the revised metric with the same shortcut

Only an interpretable measure continues

Observe the outcome and guardrails together

Retain unsuccessful tasks, missing evidence and relevant cohort differences.

Use the evidence to make a product decision

Act and preserve the definition history

Improve the product or replace a weak proxy. Do not redraw the old series under a new definition.

A metric remains useful only while its definition and guardrails reflect the user task. Preserve failed cases and definition changes when interpreting the score.

Separate diagnosis from the target

The team still needs operational measures such as search latency, indexing delay and missing-record rate. These can explain why a task failed. They should not silently replace the user outcome because they are easier to collect.

For example, a faster search can return the wrong decision sooner. A larger index can include more obsolete material. A reduction in zero-result searches can come from broadening access beyond the intended audience. Connect each input to an explicit hypothesis about how it helps the user, then inspect cases that contradict the hypothesis.

This approach leaves room for several measures without pretending they are interchangeable. The main outcome describes the intended benefit. Diagnostic measures help investigate changes. Guardrails identify damage the team is unwilling to conceal inside a positive average.

Change the metric without rewriting history

The first definition may turn out to be impractical. Perhaps “without asking another person” cannot be observed reliably, or the sample does not cover an important kind of decision. Update the measure openly.

Record the old definition, the reason for changing it, the new definition and the first comparable period. If possible, calculate both versions on a shared sample before replacing the old series. If comparison is impossible, say so instead of drawing a continuous improvement line across a definition change.

A metric review should finish with a decision about the product or the evidence. It might justify a search change, a better record-ownership process or more research with a neglected cohort. It can also conclude that the proxy is weak and should be replaced.

Before setting an ambitious target, try the easy shortcuts. A number that survives those tests is more useful for an outcome-based OKR than one that merely looks impressive on a dashboard.