← Go Back

Experimentation

What is experimentation?

Experimentation in product work means testing hypotheses with real users or real behaviour so you can learn what works instead of guessing. It includes A/B testing, usability tests, and other controlled tests. The loop: form a hypothesis, run a test, measure, learn, then decide.

Use it when: you have a debatable assumption (e.g. “this change will improve sign-up”) and you can run a test that would change your mind. Experimentation turns opinions into evidence.

Copy/paste checklist (one experiment)

  • [ ] Hypothesis: If we [change], then [metric] will [change] because [reason].
  • [ ] Method: A/B test, usability test, or other (choose what answers the question).
  • [ ] Success metric: What you’ll measure; what “success” and “failure” mean.
  • [ ] Decision rule: What you’ll do if the result is X (e.g. ship, iterate, or drop).
  • [ ] Learning captured: So the next experiment builds on it.

Why experimentation matters

  • Shifts decisions from opinion to evidence so you ship what works.
  • Reduces risk by testing with a subset before full rollout.
  • Builds a learning culture: “we test” instead of “we assume.”
  • Informs prioritisation and roadmap with real data.

What good experimentation includes

Checklist

  • [ ] Testable hypothesis – Clear “if X then Y” that can be proved or disproved.
  • [ ] Right method – A/B testing for at-scale behaviour; usability testing for why and how.
  • [ ] Defined success – Metric and threshold so you know when to act.
  • [ ] Decision in advance – What you’ll do with a positive, negative, or inconclusive result.
  • [ ] Documented outcome – So the team and future experiments benefit.

Common formats

  • A/B test: Compare control vs variant(s) on a metric; random assignment. See A/B testing.
  • Usability test: Observe users completing tasks; qualitative learning. See usability testing.
  • Smoke test / fake door: Test interest (e.g. sign-up for a feature that doesn’t exist yet). Use for demand validation.

Examples

Example (the realistic one)

Hypothesis: “If we add a one-line value prop above the sign-up form, sign-up rate will increase because users will understand the benefit first.” Method: A/B test; 50/50; primary metric sign-up rate. Success: 95% confidence and ≥2% relative lift to ship. Decision: Ship winner; if inconclusive, iterate copy and re-test. Learning: Document result and any segment differences so the next test builds on it.

Common pitfalls

  • No hypothesis: “Let’s just try it.” → Do this instead: Write “If we X, then Y will Z because…” before you run the test.
  • Wrong method: Using A/B when you need to understand why (e.g. confusion). → Do this instead: Use usability testing or user research for why; A/B for which version wins.
  • No decision rule: You get results but don’t agree what to do. → Do this instead: Define “we ship if…”, “we drop if…”, “we iterate if…” before the test.
  • Ignoring learning: Results stay in a slide deck. → Do this instead: Capture outcome and implication; share and reuse in continuous discovery and backlog.
  • Experimentation vs A/B testing: A/B testing is one type of experiment; experimentation is the broader practice (hypothesis, test, learn, decide).
  • Experimentation vs user research: User research can be part of experimentation (e.g. usability tests); experimentation emphasises hypothesis and decision.
  • Experimentation vs continuous discovery: Continuous discovery is the habit of learning; experimentation is how you test specific assumptions.

Next step

Pick one assumption (e.g. about a CTA, a flow, or a message), write it as a hypothesis, and run one experiment (A/B test or usability test). Define the decision rule before you look at results, then document what you learned.

Related Terms