Measurement
Part of How to run incrementality tests for UK marketing campaigns
How to run incrementality tests for UK marketing campaigns
How to design UK geo holdout tests, set a decision rule and collect incrementality data under UK GDPR, with a worked example priced in pounds.
What to take away
- Incrementality testing measures cause rather than correlation: you hold a comparable group back and compare the change in outcomes.
- Geo holdouts at region or postcode-cluster level are the workable UK design when person-level splitting is not available.
- Fix the decision rule, the minimum detectable effect and the end date before launch.
- UK GDPR duties apply to test data even when nothing leaves your own systems.
- Report a range. A single point estimate hides the uncertainty that decides whether a result is actionable.
Why attribution cannot answer this question
Attribution assigns credit after the event. It reads the behaviour of people who were already exposed, so it cannot say what would have happened without the campaign. A platform reporting a strong return on ad spend may be describing people who would have bought anyway.
Incrementality asks the counterfactual. You need a comparable group that did not see the campaign, measured over the same weeks with the same instrumentation. That is a test design, not a reporting format.
Tighter privacy rules have widened the gap. With less cross-site signal, platform conversion counts lean harder on modelling. Controlled incrementality exists to check those models against observed behaviour, and our guide to attribution methods sets out when it is worth the extra cost.
Designing a UK geo holdout
A geo holdout splits the market into test and control areas. Test areas see the campaign; control areas stay dark. The outcome measure is the change in each arm, compared over the same period.
Region-level splits are easy to explain to stakeholders but coarse. Scotland, Wales, Northern Ireland and the English regions trade differently, and ONS output data shows how uneven regional performance can be. A single region measured against another single region carries heavy bias risk.
Postcode clusters give finer granularity. Build clusters with similar baseline sales or enquiry volumes, then assign them at random rather than by the judgement of a regional manager.
Five steps to design the test
- Write down the decision and lift
- Choose outcome measure and capture
- Build clusters from 12 weeks baseline
- Assign clusters at random, check balance
- Fix run dates, spend and analysis
Once live, leave it alone. Topping up spend in a control area because a regional manager asks destroys the comparison.
A worked example with labelled pound figures
The figures below are illustrative. They show how the arithmetic works rather than what any particular campaign achieved.
Test vs control revenue change
Test areas
- Baseline revenue
- £180,000
- Test-period revenue
- £212,000
- Change
- +£32,000
- Change %
- +17.8%
Control areas
- Baseline revenue
- £175,000
- Test-period revenue
- £186,000
- Change
- +£11,000
- Change %
- +6.3%
Test areas
- Baseline revenue, four weeks
- £180,000
- Revenue during the test, four weeks
- £212,000
- Change
- +£32,000 (+17.8%)
Control areas
- Baseline revenue, four weeks
- £175,000
- Revenue during the test, four weeks
- £186,000
- Change
- +£11,000 (+6.3%)
The incremental figure is the difference in change between the arms, applied to the test baseline. Test revenue rose 17.8 per cent; control revenue rose 6.3 per cent. That 11.5 point gap on a £180,000 four-week baseline is about £20,700.
Against £14,000 of media spend in the test areas, contribution is roughly £6,700, or about £1.48 returned per pound spent. Quote the interval around it. A range that straddles zero has not settled anything.
Collecting test data under UK GDPR
A holdout rarely needs less data than a normal campaign, and the rules do not relax because the purpose is measurement. If you build clusters from customer records or match outcomes back to individuals, you are processing personal data.
Record a lawful basis under the UK GDPR, keep cluster inputs to the minimum you need and set a retention period before launch. The ICO's UK GDPR guidance and resources cover lawful basis, transparency and retention wording.
Aggregate outcomes before they leave the test platform if you can. If the design relies on server-side collection rather than browser tags, read our checklist on server-side tracking under UK GDPR before you size the test.
Common questions
How long should a UK incrementality test run?
Long enough to cover a full purchase cycle for the product. Four weeks is a common minimum for shorter cycles, but a considered purchase may need eight to twelve weeks before the estimate stabilises.
Can a small UK advertiser run a geo experiment?
Yes, but expect wide intervals. Match clusters on baseline volume, accept fewer clusters and treat the result as directional rather than precise.
Do I need consent to run a holdout?
Not always. Consent is one lawful basis among several. What you cannot skip is a documented basis, data minimisation and a retention rule.
