← Go Back

Experimentation Patterns

What are Experimentation Patterns?

Experimentation patterns are systematic approaches to testing hypotheses about user behaviour, feature effectiveness, and product improvements through controlled experiments (e.g. A/B tests). They help you validate assumptions and make decisions based on evidence rather than opinion.

Use it when: you have a clear hypothesis about a change (copy, layout, flow) and want to learn from real user behaviour before committing fully.

Copy/paste template

  • Hypothesis: [If we X, then Y will improve because Z.]
  • Primary metric: [what you’re optimising, e.g. sign-up rate]
  • Secondary metrics: [guardrails, e.g. retention, time on task]
  • Variants: [control vs treatment(s), clearly defined]
  • Sample size / duration: [enough for significance; avoid stopping early]
  • Success criteria: [what would make you ship or roll back]

Why Experimentation Patterns matter

  • Replace guesswork with evidence so you ship what actually works.
  • Reduce risk by testing with a subset of users before full rollout.
  • Build a culture of learning and data-informed decision making.
  • Focus effort on changes that move the needle on key metrics.
  • Create a shared language for "we believe X; let’s test it."

What a good experiment includes

Checklist

  • [ ] Clear, testable hypothesis (if X, then Y; measurable).
  • [ ] Single variable (or controlled multivariate design) so you know what drove the result.
  • [ ] Adequate sample size and duration for statistical reliability.
  • [ ] Primary and guardrail metrics so you don’t optimise one thing at the expense of others.
  • [ ] Pre-registered plan (what you’re testing, when you’ll look, what you’ll do) to avoid cherry-picking.

Common formats

  • A/B test: one control, one treatment; compare on a primary metric.
  • Multivariate / MVT: test multiple factors; use when you have traffic and need to understand interactions.

Examples

Example (the realistic one)

Hypothesis: "If we shorten the sign-up form from 8 fields to 4, completion rate will increase without hurting quality." They run an A/B test, primary metric = completion rate, guardrail = lead quality (e.g. conversion to paid). After a pre-set sample size, the short form wins on completion and quality is unchanged. They ship the short form.

Common pitfalls

  • Testing without a hypothesis: running tests to "see what happens". → Do this instead: write a hypothesis and success criteria before you start.
  • Stopping early or peeking: deciding based on incomplete data. → Do this instead: pre-define sample size/duration and stick to it, or use sequential testing methods correctly.
  • Changing too many things: you can’t tell what caused the result. → Do this instead: test one change (or a clearly defined set) per experiment.
  • Ignoring guardrail metrics: optimising one metric while harming another. → Do this instead: define primary and guardrail metrics; don’t ship if guardrails regress.
  • Experimentation vs A/B testing: A/B testing is one experimentation pattern; experimentation patterns also include design of experiments, metrics, and decision rules.
  • Experimentation vs user research: Experiments measure behaviour at scale; user research (e.g. interviews) explains why. Use both: research to form hypotheses, experiments to test them.
  • Experimentation vs gut feel: Experimentation uses data and structure; gut feel is unvalidated. Use experiments when the stakes or uncertainty are high.

Next step

If you’re new to running experiments, start with A/B testing and define one clear hypothesis and metric. If you’re forming hypotheses from user insight, use user research and usability testing.

Related Terms