Work

Three full case studies and five shorter ones. All from prior employment at a publicly traded two-sided marketplace for high-value physical goods, anonymized. Every number was reported to the executive who asked for it.

Turning a margin-negative pilot into a seven-figure delivery program

Client
A publicly traded two-sided marketplace for high-value physical goods
Role
Pod lead; owner of the analysis
Offer this maps to
B — ML Decision Model
  • 3%was about 0.4%take rate on eligible listings, about 14 months

  • 15%was 5%programmed margin

Situation

The marketplace piloted a new long-distance delivery program in one region. Instead of legacy brokers, it priced transport through an API-connected partner. The pilot was built to grow adoption, so margins were set thin on purpose.

Leadership had two questions before going nationwide. Did the program grow adoption among buyers? And did the economics work, or would scaling it scale a loss?

Nobody had a clean answer. Take rates were available, but not by buyer segment or against comparable fulfillment sites outside the pilot. Nobody had built a P&L for the program or modeled the margin needed to break even. The decision had a C-level sponsor and a fixed executive readout date.

What we did

The pod lead owned the analysis end to end.

Adoption. Compared take rates for long-distance delivery at pilot fulfillment sites against non-pilot sites over the same period. Cut adoption by buyer tenure and buyer segment to see who was using it and who was not.

Liquidity. Measured the marketplace attributes of delivered listings — offers, price realization and cycle time — against listings that were not delivered, to check whether the program changed how items sold, not just how they shipped.

Economics. Built a pro-forma P&L for the program: transport cost through the partner, programmed margin, storage-revenue effects, and cycle-time effects. Then ran break-even scenarios at several margin levels, so leadership could see the margin that cleared zero and the margin that paid for the program.

Readout. Presented twice to the C-level sponsor ahead of the executive readout, so the sponsor could challenge assumptions before the meeting rather than during it.

Follow-up. After the decision, ran a delivery-estimate analysis and a lookback on margin and adoption to check whether the model held up in the field.

Result

Leadership approved nationwide rollout.

The analysis showed the pilot was margin-negative as configured. The break-even model gave the margin needed to fix it. Programmed margin was raised from 5% to 15%. Effective margin moved from under 1% to about 11%.

Adoption held. Long-distance take rate grew from about 0.4% to about 3% of eligible listings in about 14 months. Buyers saved roughly 40% on average delivery cost against the legacy alternative, so the margin change did not price them out.

Projected annual gross impact was seven figures — roughly 2.5 to 3 times the prior model. The pilot period itself closed at a small net loss. The post-adjustment period was strongly positive.

One limit, stated at the time: the pilot ran in a single region, so the nationwide projection assumed take rates would behave similarly in other markets. The lookback after rollout was built to test that assumption rather than take it on faith.

What it means for you

If you have a program, a pricing change or a channel that is growing but nobody can say whether it makes money, this is the shape of the work. One decision. A model of the economics with a break-even. A readout the sponsor has already seen twice. The ML Decision Model offer (6–10 weeks, fixed scope) is built for exactly this question.

Methods used

  • Treatment vs. control comparison across pilot and non-pilot fulfillment sites
  • Segment cuts by buyer tenure and buyer type
  • Liquidity analysis of delivered vs. non-delivered listings
  • Pro-forma P&L with storage-revenue and cycle-time effects
  • Break-even margin scenario modeling
  • Post-rollout lookback analysis
  • SQL, Python, Tableau

Open this case study on its own page


Proving a redesign worked when the top-line metric said it didn't

Client
A publicly traded two-sided marketplace for high-value physical goods
Role
Pod lead; measurement owner; A/B test sign-off
Offer this maps to
A — Fractional Head of Product Analytics
  • 32%was about 48%search-results exit rate, desktop

  • 25%shortertest durations with CUPED, across 30+ tests

Situation

Three major redesigns were launching close together: member registration, the homepage and the listing detail page. Leadership wanted proof, not anecdotes, before any of them went to 100% of users.

Then the listing detail page test showed a rise in exit rate. Exit rate was the metric everyone watched. The read in the room was that the new page was losing users. A rollback was on the table.

At the same time, a drop in unique users appeared in the data with no obvious cause. Two red flags on one test made the decision harder than the design itself.

What we did

KPI frameworks first. For each redesign, defined the primary metric, the guardrail metrics and the segments before the test started. Registration was measured on clicks to register, activation rate and payment-to-activation time. The listing page was measured on exits, bounce and downstream engagement, by device.

Instrumentation. Set up the A/B tests in Statsig and built Fullstory dashboards for the test segments, so product and design could watch the same numbers the analysts were reading. Wrote the test query from scratch rather than inheriting a legacy definition.

The exit-rate deep dive. Broke exits down by the page users were on when they left. The question was not "did exit rate go up" but "where did exits move from and to."

The unique-users drop. Traced it to mobile-web deep-linking: users arriving through deep links were landing differently on the new page and were being counted differently. It was a tracking artifact, not a loss.

Sign-off. Led the A/B sign-off discussions with product leadership, with the readout in the standard format: hypothesis, exposure, duration and sample, primary and guardrail metrics with confidence intervals, diagnostics, decision.

Result

The higher exit rate on the listing page was a funnel shift, not lost users. Users were finishing on the listing page instead of bouncing back to search results or the category page.

  • Search-results exits fell from about 48% to 32% on desktop and from about 57% to 48% on mobile.
  • Category-page exits fell from about 44% to 24% on desktop and from about 47% to 39% on mobile.
  • Bounce was flat.

The listing page rolled out to 100% of users.

The registration redesign had cleaner numbers. Clicks to register fell from about 35 to about 18. Activation rose from under 80% to over 83%, a gain of more than 3 points. Payment-to-activation time fell about 18%. Product scaled it to all members. The impact was roughly 300 additional activated members per month and a seven-figure annual revenue opportunity.

The program behind the tests. Across the role, the pod ran 30+ A/B tests. Introducing CUPED — using pre-experiment behavior as a covariate to reduce variance — cut test durations by about 25%. Shorter tests meant more decisions per quarter from the same traffic.

What it means for you

A test result is only as good as the person reading it. A top-line metric moving the wrong way does not tell you why, and a rollback based on it can throw away a good redesign. The Fractional Head of Product Analytics retainer puts a senior reader on every test: metric definitions before launch, the right segments and diagnostics during, and a sign-off readout your executives will trust after. Two days a week. Three-month minimum.

Methods used

  • KPI framework and guardrail definition per test
  • A/B test setup in Statsig; segment dashboards in Fullstory
  • Exit-rate decomposition by page and device
  • Funnel-shift diagnosis
  • Tracking-artifact root cause (mobile-web deep-linking)
  • CUPED variance reduction
  • Standard readout: hypothesis → change → exposure → duration and sample → primary and guardrail metrics with CIs → diagnostics → decision → next step
  • SQL, Python

Open this case study on its own page


Replacing lossy tagging with a digital-experience platform in six months

Client
A publicly traded two-sided marketplace for high-value physical goods
Role
Pod lead; owner of the business case and the renewal case
Offer this maps to
A — Fractional Head of Product Analytics (instrumentation ownership inside the retainer)
  • 1 daywas about 3 daysto resolve a high-priority issue

  • 9+teamsadopted within about 6 months

Situation

Product analytics ran on GA4 with manual tagging through a tag manager. Three problems had compounded.

The data was lossy. An estimated 15–20% of data was never captured, because every event depended on a tag someone had to write, deploy and maintain.

Issue detection was slow. When a page broke or a flow regressed, the team found out from complaints or from a weekly review, not from the data.

Engineering was paying for it. Engineers spent an estimated 5–10% of each feature's bandwidth on tagging and QA. Every feature shipped slower so the analytics could keep up.

The fix — a DOM-capture digital-experience platform that records everything without per-event tags — needed capital approval, a vendor decision and a rollout across teams that had never worked from one dataset.

What we did

Vendor evaluation. Compared two DOM-capture vendors on features and cost against a written requirements list: session capture, funnel and segment analysis, A/B test segment support, mobile coverage, data export, and the fit with the existing BigQuery and Statsig stack.

Business case. Built the capital-approval case around a validated revenue-opportunity figure rather than a feature list. Defined the ROI KPIs leadership would check at renewal: data captured, time to detect and resolve issues, engineering bandwidth returned, and teams actively using the tool. The case identified a high-six-figure annual revenue opportunity.

Rollout. Led the rollout through an analyst on the pod. Built P0 metrics dashboards first, so leadership saw value in the first weeks. Then A/B test segments, so experimentation moved onto the new platform. Then onboarding sessions for product, design, QA, mobile, seller and marketing teams, each with a dashboard built for that team's questions.

Renewal. Built the renewal case against the same KPIs defined in the business case. The team used it to negotiate contract concessions.

Result

The platform was live within about 6 months and adopted by 9 or more teams.

High-priority issue resolution fell from about 3 days to about 1 day.

One early catch paid for much of the rollout. The platform detected a +45% page-load-time regression caused by a bot-check script. Root cause was found in the session data and the fix shipped within 2 days of it.

The platform now powers the A/B readouts for search, the listing page and registration — the tests described in the other case studies.

At the program level, the behavioral dashboards and models built on this foundation contributed to 12% higher engagement and 18% lower search abandonment.

What it did not do: it did not remove the need for a tracking plan. DOM capture records everything, but someone still has to define the events that matter, name them, and own them. That ownership sat with the pod.

What it means for you

If you have bought a product analytics tool and it is not earning its contract, the problem is usually ownership, not the tool. Someone senior has to write the requirements, build the business case in revenue terms, roll it out team by team with a dashboard each team actually uses, and hold the vendor to KPIs at renewal. Inside the Fractional Head of Product Analytics retainer, that ownership is part of the job. It is not sold as a separate project unless you ask for one.

Methods used

  • Written requirements and two-vendor comparison on features and cost
  • Capital-approval business case with validated revenue opportunity and ROI KPIs
  • Phased rollout: P0 metrics → A/B segments → team-by-team onboarding
  • Issue detection and root-cause analysis from session data
  • Renewal case built on the original KPIs
  • Fullstory, GA4, Statsig, BigQuery, Tableau

Open this case study on its own page

Five shorter cases

AI search ranking: the true effect, and a cannibalization check

Client
A publicly traded two-sided marketplace for high-value physical goods
Role
Pod lead, analysis owner

Keyword search was replaced with AI-personalized ranking. The A/B result was positive overall, but one buyer segment was buying more at lower prices, and leadership wanted to know whether personalization was pushing cheap listings to one buyer group at the expense of another.

The pod ran a 5-Whys review on that segment's result, modeled incremental revenue from buying activity, price and price realization, and ran a low-value-listing check comparing each segment's share of low-value listings before and after. It also delivered click-position metrics so product could see how ranking changes moved behavior.

Incremental revenue modeled in the low-to-high single-digit millions. The cannibalization hypothesis was disproved: the segment's share of low-value listings fell about 3 percentage points. AI search scaled to 100% of users. In year two the pod led the renewal scenario analysis for a large multi-year vendor renewal using a disciplined framework rather than vendor-supplied ROI.

Member-level customer lifetime value

Client
A publicly traded two-sided marketplace for high-value physical goods
Role
Pod lead, model owner

Retention and paid-media decisions were being made on counts of members, not on what each member was worth. The pod designed a member-level CLV model: annual value multiplied by expected lifespan.

Annual value came from each member's incremental offers above competing buyers, weighted by listing-type liquidity and fee ratio, plus fees paid. Expected lifespan came from an XGBoost churn model using tenure, listings engaged with, logins, watchlists and days since last activity. The model was delivered monthly through Tableau and Excel so marketing and product used the same numbers.

The result was a member-level CLV with segment benchmarks and churn-risk flags. It became the foundation for value-based retention, segment prioritization and paid-media ROI. The pod co-authored the scoping for a paid-media CLV extension with an external modeling partner and secured executive sponsorship for it.

A pickup KPI the whole company adopted

Client
A publicly traded two-sided marketplace for high-value physical goods
Role
Pod lead, analysis owner

Buyer pickup at physical fulfillment sites was a known pain point with no agreed metric. The pod analyzed about 900,000 pickups: punctuality, wait and cycle times, top and bottom 10 sites, appointment window sizes and loader productivity. It defined a new appointment-honor-rate metric, ran a root-cause review at one underperforming site including a site visit, and drafted a forecasting tool concept.

Findings: average cycle time was about 40 minutes; over 25% of appointments took an hour or longer; only about 40% of high-volume logistics partners consistently got consecutive appointments; sub-one-hour windows drove congestion; and throughput depended more on process than headcount.

The "Pickup Window Honored %" KPI was adopted into the general-manager dashboard company-wide. Operations got one number to manage to.

AI classification of 70,000 customer contacts

Client
A publicly traded two-sided marketplace for high-value physical goods
Role
Pod lead; later analytics owner of the productionized report

Customer success knew call volume was high. Nobody knew why in a form product could act on. The pod applied AI classification to about 70,000 calls to derive roughly 100 contact reasons, then built a three-tier taxonomy with product and customer success and mapped each reason to a journey touchpoint.

The work was later validated and productionized across voice, chat and WhatsApp: about 300,000 conversations, about 500,000 tags, in 7 categories, about 40 sub-categories and about 100 reasons.

About 95% of contacts were found to be preventable. Top drivers were ownership-paperwork status (about 25%), payment rejections (about 20%) and buyer eligibility (about 10%). The pod recommended chatbot-first routing for specific topics and an account-level status page, and identified about 5,000 members making five or more contacts.

Cross-listing partnership: attribution before the partner's data arrived

Client
A publicly traded two-sided marketplace for high-value physical goods and a subsidiary specialty marketplace
Role
Lead analyst

Listings were cross-listed between the marketplace and the subsidiary. Leadership needed to know whether the partnership lifted price and engagement, or just moved the same buyers around.

The analyst combined data from both platforms on about 50,000 cross-listed listings and built attribution logic. The engagement funnel ran from about 1 million visitors to roughly 13,000 registrations, 6,000 buyers and 1,000 completed purchases. Before-and-after comparisons were adjusted for expected growth, with difference-in-differences style exhibits. While waiting for partner data, category prices were scraped so the analysis did not stall.

Results since launch: price up about 2%, buyers per listing up about 3 to 4%, offers per listing up about 13%. Cross-listed categories grew about 12% against a decline of about 2% for non-listed. Registration conversion of about 1% was flagged with UX guardrails. Leadership approved continuing the partnership with filters, tracked on weekly and monthly partner KPIs.

Public ML demo: pricing optimization on a two-sided marketplace

A worked example of the ML Decision Model offer on synthetic data: demand model, pricing policy, evaluation against a hold-out, and a readout. MIT licensed, so you can run it and read the code before hiring anyone.

Repository on GitHub [TODO: GitHub URL]

Book a free 30-minute Decision Audit