The quiet rise of causal inference in product analytics

Every product team eventually hits the same wall: the change they want to measure can't be randomized. You can't A/B-test a price increase for a specific customer segment. You can't randomize a marketing channel that already launched. You can't run an experiment retroactively. So you do the next-best thing — and most teams do it badly.

The randomization gap

The instinct, when you can't randomize, is to compare. Sales before vs after. Treated customers vs untreated. The problem is that almost every comparison like that is contaminated by selection effects, time effects, or both. The numbers look meaningful. They aren't.

The practical toolkit

Causal inference gives you a handful of techniques to recover something close to what an A/B test would have told you — provided you understand the assumptions.

• Difference-in-differences: compare changes in treated and untreated groups over the same window.
• Synthetic control: build a weighted combination of unaffected units that mimics the treated unit's pre-period.
• Regression discontinuity: when treatment is assigned by a sharp threshold, units just above and just below are nearly randomized.
• Propensity score matching: pair treated and untreated units that look identical on the variables that predicted treatment.

When not to use it

If you can randomize, randomize. If your treatment effect is small relative to the noise in your data, no causal method will save you. And if you can't articulate a plausible story for why the treated and control groups would have evolved similarly absent the treatment, you don't have a causal study — you have a comparison with extra math.

Used well, causal inference unlocks the decisions that A/B tests can't reach. Used badly, it gives bad analyses a veneer of rigor. The discipline is knowing which one you're doing.