The Institute of Clever StuffStart your Discovery

The plumbing · Data engineering and architecture

Four years building the backbone: a single view of 16.2m customers

Client

Global energy business, customers and products division, four markets

Challenge

Every team building its own version of the truth. Data products that worked, but only through heroic effort around foundational gaps.

Result

A single customer view covering 16.2m UK customers, the first cross-brand B2B intelligence, and foundations that every later product was built on.

The situation

The business had invested in modern infrastructure and could build valuable data products. But each one required heroic individual effort to work around what the foundations couldn't do. Data was duplicated across business lines and corrected at the hub rather than at source. Ownership was unclear, stewardship murky, and every new use case cost as much as the last.

That pattern is common, and it's expensive: every use case built on weak foundations creates technical debt, not business value.

What we built

Over four years embedded across four markets, we engineered from the plumbing up.

Data engineering: pipelines and ETL across the estate. Platform and tooling on AWS and Databricks. Data governance frameworks and standards, and data privacy compliance to GDPR and beyond, including enterprise consent controls. Domain architecture for customer and product. Data models and ontology giving business lines common structures where each had previously kept its own.

On top of that backbone, the single customer view: stitching disparate data sources together into one view of 16.2m UK customers, then extending the approach to B2B to create the first cross-brand intelligence the business had.

In parallel work for a global retailer, the same discipline took a different shape: an automated data ingest framework, a single cloud direction, provisioning of source data for every analytics product, and a fully working CI/CD pipeline, a reusable foundation team that every product colony drew on, so each new product cost less than the one before.

Why it matters

The single customer view proved the principle. Value doesn't come from any one data source; it comes from the model that joins them. Get the ontology and common models right and the value compounds. Get them wrong and every business line builds its own version of the truth, and you pay for the same problem repeatedly.

Every product built afterwards, site asset optimisation, EV planning, the transition modelling, stood on foundations that had already been laid.

What we took from it

Fix the backbone once and accelerate everything above it. The hardest part is rarely the engineering; it's the ownership, stewardship and accountability for quality at source that keep the backbone from decaying after it's built.

What's the biggest opportunity in your business?

Take the assessment. Get a tailored Discovery Blueprint, generated for you.