Proving a redesign worked when the top-line metric said it didn't

Client
A publicly traded two-sided marketplace for high-value physical goods
Role
Pod lead; measurement owner; A/B test sign-off
Offer this maps to
A — Fractional Head of Product Analytics

Situation

Three major redesigns were launching close together: member registration, the homepage and the listing detail page. Leadership wanted proof, not anecdotes, before any of them went to 100% of users.

Then the listing detail page test showed a rise in exit rate. Exit rate was the metric everyone watched. The read in the room was that the new page was losing users. A rollback was on the table.

At the same time, a drop in unique users appeared in the data with no obvious cause. Two red flags on one test made the decision harder than the design itself.

What we did

KPI frameworks first. For each redesign, defined the primary metric, the guardrail metrics and the segments before the test started. Registration was measured on clicks to register, activation rate and payment-to-activation time. The listing page was measured on exits, bounce and downstream engagement, by device.

Instrumentation. Set up the A/B tests in Statsig and built Fullstory dashboards for the test segments, so product and design could watch the same numbers the analysts were reading. Wrote the test query from scratch rather than inheriting a legacy definition.

The exit-rate deep dive. Broke exits down by the page users were on when they left. The question was not "did exit rate go up" but "where did exits move from and to."

The unique-users drop. Traced it to mobile-web deep-linking: users arriving through deep links were landing differently on the new page and were being counted differently. It was a tracking artifact, not a loss.

Sign-off. Led the A/B sign-off discussions with product leadership, with the readout in the standard format: hypothesis, exposure, duration and sample, primary and guardrail metrics with confidence intervals, diagnostics, decision.

Result

The higher exit rate on the listing page was a funnel shift, not lost users. Users were finishing on the listing page instead of bouncing back to search results or the category page.

The listing page rolled out to 100% of users.

The registration redesign had cleaner numbers. Clicks to register fell from about 35 to about 18. Activation rose from under 80% to over 83%, a gain of more than 3 points. Payment-to-activation time fell about 18%. Product scaled it to all members. The impact was roughly 300 additional activated members per month and a seven-figure annual revenue opportunity.

The program behind the tests. Across the role, the pod ran 30+ A/B tests. Introducing CUPED — using pre-experiment behavior as a covariate to reduce variance — cut test durations by about 25%. Shorter tests meant more decisions per quarter from the same traffic.

What it means for you

A test result is only as good as the person reading it. A top-line metric moving the wrong way does not tell you why, and a rollback based on it can throw away a good redesign. The Fractional Head of Product Analytics retainer puts a senior reader on every test: metric definitions before launch, the right segments and diagnostics during, and a sign-off readout your executives will trust after. Two days a week. Three-month minimum.

Methods used

Book a free 30-minute Decision Audit