← Back to framework
Tools / Bucket 6

Testing and optimisation

A/B tests, holdouts, geo tests. AI runs them faster. You form hypotheses worth testing and know when results are noise.

Tools and platforms

/A/B testing (built into most platforms)

One variable, two versions, measure the difference. Meta, Google, and email platforms all have native A/B testing. The mechanics are automated. The hypothesis is yours.

/Google Optimize (sunset) / VWO / Optimizely

Website A/B and multivariate testing tools. Test different headlines, layouts, CTAs on your site. VWO and Optimizely are the main options now that Google Optimize has been discontinued.

vwo.com →
/Holdout testing (manual setup)

Exclude a control group from seeing ads. Compare their behaviour to the exposed group. The only way to measure true incrementality. Requires discipline: you are deliberately not showing ads to people who might buy.

/Geo testing

Run different strategies in different regions. Useful when you cannot do individual-level holdouts. Compare Dublin vs Cork, or UK vs Ireland, and measure the difference.

What AI does vs what you do

AI handles
  • Traffic allocation across variants
  • Run dozens of variants simultaneously
  • Statistical significance calculation
  • Auto-optimise by shifting budget to winners mid-test
  • Pattern detection across test results
You decide
  • Form the hypothesis (what do you expect, who does it affect, and why)
  • Decide what is worth testing in the first place
  • Interpret results honestly (not just pick the number you like)
  • Spot noise vs signal (small early swings are not results)
  • Know when a result generalises and when it does not

Skills to develop

/Hypothesis formation
/Sample size and timing judgment
/Statistical significance interpretation
/Novelty effect recognition (the emoji subject line effect fades in 6 weeks)
/Result skepticism
/Metric alignment (test clicks but measure purchases separately)

A test result tells you what happened in that test, with that audience, at that time. It does not automatically generalise. The skill is knowing which question is worth asking before you run anything.

← Marketing technology
Planning and budgeting →