Operations
Part of Running measurement operations with five gates that cannot substitute for one another
Privacy first measurement quality checklist: release evidence for one metric
A privacy first measurement quality checklist for one metric, such as a newsletter signup: definitions, device storage, tests, access and hold rules.
A privacy first measurement quality checklist is a release gate for one metric, such as a newsletter signup. Each item asks for retained evidence tied to the exact version under review, and a check passes only when its named owner can point to that evidence.
These checks cover collection, processing, reporting and any business decision they could block. They form a non-ranked sequence, separate from the wider measurement operations process. Qualified privacy, legal, security, accessibility and measurement reviewers must adapt them to the service.
What to take away
- A release gate needs retained evidence for the version under review, not a compliance badge.
- Every check needs an owner, an observable result and a defined response to failure.
- Passing a small functional test does not prove production completeness or business value.
Fix the signup definition before anything runs
Start by writing down what a signup is. The ticket names one decision and the person authorised to make it, and the metric states its unit, population, period, time zone, numerator and denominator where relevant.
- The ticket names one decision and the person authorised to make it.
- Every measure has a unit, population, period, time zone, numerator and denominator where relevant.
- Inclusion, exclusion, deduplication and attribution rules are versioned.
- The report states what it cannot establish, including causal limits.
Evidence should include a metric dictionary, a sample calculation and a decision note. A vague label such as "engagement" fails until its observable inputs and permitted interpretation are defined, and a signup is no different.
An illustrative pack for the signup holds the versioned dictionary entry and the synthetic journey script. It also holds the reconciliation query and result, the build identifier, the tester name and the decision note. That pack shows which build the owner approved.
Purpose and definitions checks
- Name one decision and authorised owner
- Define unit, population, period, time zone
- Version inclusion, exclusion, deduplication, attribution rules
- State what report cannot establish
Check what the signup form stores and sends
Inventory the device storage or access operations across acceptance, refusal and withdrawal. Do not let a vendor category decide whether a check applies. A privacy reviewer owns the conclusion, while the tester supplies factual behaviour.
- Purposes, fields, identifiers, recipients, retention and transfers are mapped.
- Actual controller and processor responsibilities are documented.
- Lawful basis, transparency and DPIA decisions under UK GDPR cite the reviewed version.
- Device storage or access operations covered by PECR are inventoried across acceptance, refusal and withdrawal.
The ICO's storage and access technologies guidance explains the exceptions to the prohibition on storing or accessing information on people's devices. Record the data protection decisions against the same version the testers ran, not an earlier one.
Run the journey and reconcile the result
Run a known synthetic signup and compare what the pipeline recorded with what should have happened. Retain the test input, expected output, actual output, environment, build version and tester, so a reviewer can repeat the run.
- A known synthetic journey produces the expected events, values and order.
- Refused and withdrawn states produce only the authorised behaviour.
- Duplicate, interrupted, delayed and invalid journeys have expected outcomes.
- Aggregate output is reconciled to an independent operational source where available.
- Differences are quantified and explained against an approved tolerance.
Passing a small functional case does not prove production completeness. A tolerance pass does not establish incrementality or business value. Say so in the decision note, so the signup metric is not used beyond what the evidence supports.
Functional and accuracy tests
- Synthetic journey produces expected events
- Refused and withdrawn states behave correctly
- Duplicate, interrupted, delayed journeys have expected outcomes
- Reconcile aggregates to independent source
- Quantify differences against approved tolerance
Security and incident checks for the signup data
Give human and service identities separate, least-privilege access, and make privileged changes and relevant access generate protected logs. The security lead should test the local implementation rather than accept a feature description.
- Human and service identities have separate, least-privilege access.
- Privileged changes and relevant access generate protected logs.
- Supplier and internal emergency contacts have been verified.
- Backup, recovery and credential-revocation procedures have owners.
- The incident route distinguishes security events from confirmed personal-data breaches.
The NCSC advises SaaS customers to protect administration, monitor activity and prepare for incidents in its secure-use guidance, Using SaaS securely. That page sits in the Using cloud services securely collection.
Accessibility evidence and the equality-law line
Reviewers must be able to use the dashboard and the evidence pack. GOV.UK's service manual tells teams to test with both automated and manual testing, so record manual checks of controls, focus order, labels and status messages.
- Dashboard controls, focus order, labels and status messages have manual checks.
- Data tables and charts have an equivalent readable presentation.
- Keyboard and agreed assistive-technology journeys are recorded.
- Evidence packs use clear headings, meaningful link text and identifiable errors.
The cited ICO and NCSC guidance applies across the UK, so collection and security checks do not change at the border. The main legal difference sits in equality law: the Equality Act 2010 applies in England, while Northern Ireland uses separate equality legislation. Obtain qualified advice for the organisation's duties and user needs.
Decide: pass, narrow or hold
A release either has the evidence required for its stated scope or needs a narrower scope, remediation or further review. Do not average failures into a score. Any unapproved field, permission-state failure, unexplained material variance or loss of critical logging should return a hold verdict.
- All approvals reference the candidate version and disclose open conditions.
- Monitoring has thresholds, observation windows and responders.
- The team has rehearsed collection stop, report suppression and rollback separately.
- Exceptions name scope, mitigation, owner, authority and expiry.
- A post-release review trigger is scheduled.
Common questions
What goes in the evidence pack for a newsletter signup?
The pack holds the versioned dictionary entry and the synthetic journey script, plus the reconciliation query and result. It also holds the build identifier, the tester name and the decision note, so a reviewer can see which build was approved and by whom.
Does a passed test show the signup metric is right in production?
No. Passing a small functional case does not prove production completeness, and a tolerance pass does not establish incrementality or business value. The decision note should say what the evidence supports and what it does not, so the metric is not used beyond that.
Does the checklist change outside England?
The ICO and NCSC guidance applies across the UK, so the collection and security checks stay the same. The equality-law check differs: the Equality Act 2010 applies in England, while Northern Ireland uses separate equality legislation. Qualified advice should settle the organisation's duties.
