Segmentation Without Self-Deception

Segmentation tells stories the average hides. A flat overall result can hide a +12% lift on mobile and a -10% loss on desktop. Without segments, you would have called this a loser.

But segmentation is dangerous. Each cut multiplies your chance of finding a significant result by chance alone. The fix is to pre-declare which segments you will analyse, before the test launches, in your hypothesis document.

Three segments is usually enough: device, source, returning vs new. Anything more and you are fishing. Anything less and you are blind.

// Pre-declare segments in your test plan.
const segments = ['device', 'channel', 'visitor_type'];
segments.forEach((s) => report(s));