Measurement
Part of Privacy-first measurement: measurement register template
Six stop conditions for a privacy first measurement report, and how to re-release
Six reasons to withhold a privacy first measurement report in England, the repair that clears each one, and how to log the correction before re-release.
A privacy first measurement report should stop before release when one of six conditions holds. Each condition has a repair that clears it and an owner who confirms the repair. Stopping is not failure. It keeps a wrong number from turning into a decision.
What to take away
- Withhold a rate when the denominator is absent. Publish a count or nothing.
- Stop when an unapproved device operation occurs or a proxy materially changes the result.
- Stop when uncertainty crosses the decision threshold or the revision impact is unknown.
- Stop when a causal claim lacks an adequate comparison, and say credited, not generated.
- Keep the failed release visible and log its definition, affected period, original decision and correction.
Read the six stop conditions
A stop condition is an observable fact, not a feeling about quality. The table pairs each condition with the repair that clears it and a suggested owner. A report restarts only when its row is cleared.
| Stop condition | Repair that clears it | Suggested owner |
|---|---|---|
| Denominator is absent | Publish a count or withhold the rate until eligible starts are available | Data owner |
| Unapproved device operation occurs | Approve or remove the operation, then re-test permission behaviour | Privacy owner |
| A proxy materially changes the result | Validate the population, proxy and decision use again | Named assurer |
| Uncertainty crosses the decision threshold | Place the limitation beside the value and re-ask whether it can carry the decision | Decision owner |
| Revision impact is unknown | Establish whether the change is scheduled or corrects an error, then size its effect | Data owner |
| A causal claim lacks an adequate comparison | Replace "generated" with "credited" | Qualified statistician |
Every repair needs a date as well as an owner: the date the report becomes usable again. Without it, a stop becomes indefinite shelving. The metric card that holds the denominator rule is built in the measurement register template.
Stop on the denominator and the collection boundary
A rate needs a base. A dashboard may label completed events "conversion rate" without one. If eligible starts are unavailable, publish a count or withhold the rate. Both are honest, and a rate with no base is not.
Collection gaps stop a report for a different reason. Recorded events may exclude refusal, blocked scripts, offline activity or broken devices, and permission behaviour is a collection boundary, not a nuisance variable to erase. Do not inflate the observed group to the entire England audience without a model and evidence.
The ICO's storage and access guidance requires analysis of actual device operations and purposes. An operation nobody approved therefore halts the report until it is approved or removed.
Stop on validity and uncertainty
A query can match its specification and still answer the wrong question. Verify transformations with known inputs, then validate the population, proxy and decision use. Record both results and the assurer.
The AQuA Book says analytical quality assurance is more than checking the analysis against its specification (verification) and should also check that it is fit for its purpose (validation). A private team can apply that distinction without claiming government sign-off.
A provisional value may move when late data arrive, and a model may have a wide sensitivity range. Put the limitation beside the affected value. When uncertainty crosses the decision threshold, the report is no longer fit to carry that decision.
Stop on revisions and causal claims
Unknown revision impact is a stop because readers cannot tell a stable figure from a provisional one. The Government Analysis Function separates scheduled revisions from unscheduled corrections after errors, and says users should be told as soon as practicable.
The same guidance points to explaining the scale, nature, cause and impact of a change alongside the figure, so a reader can judge how far to trust the corrected value.
Credit and cause need the same care. Last interaction and modelled attribution allocate outcomes among eligible touchpoints. They do not necessarily estimate the counterfactual outcome without the intervention.
Replace "generated" with "credited" unless a suitable causal evaluation, such as an incrementality test, supports the stronger word. A qualified statistician must judge whether the comparison is credible for the commercial setting.
Log the failure and re-release
Keep the failed release visible after repair. Record these five items in the revision log so later readers do not rebuild a time series from incompatible values.
- The definition that applied to the failed release.
- The affected period.
- The decision taken on the original figure.
- The correction made.
- The owner and the date the report became usable again.
A high reconciliation result does not show timeliness, accessibility, comparability or correct personal information. Review quality by dimension and attach a consequence to each failure before the report goes out again.
Before you act
- Confirm a denominator exists before labelling any count a rate.
- Check every device operation against the approved list before release.
- Validate population, proxy and decision use after verifying the code.
- Place uncertainty and revision notes beside the affected values.
- Use credited instead of generated without a causal evaluation.
- Write the revision log entry before the corrected report is published.
Common questions
What should happen when eligible starts are unavailable?
Publish a count or withhold the rate. A rate without a documented denominator, population, period and unit should not be published as a rate.
Can attributed credit be called incremental?
Only when a suitable causal evaluation supports the stronger word. Last interaction and modelled attribution allocate outcomes among eligible touchpoints, so replace generated with credited unless that evidence exists.
What belongs in the revision log?
The definition, the affected period, the original decision and the correction. Keep the failed release visible after repair so later readers do not rebuild a time series from incompatible values.
