For years, we strived for a distributed approach to solve this by decentralizing ownership, so domain teams (like "Payments" or "Logistics") could manage their own data products. A centralized governance structure is crucial in this setup to ensure a manageable load on teams and prevent business risk. Additionally, without a unified layer to define truth, we face the metric divergence challenge: Why does the Marketing dashboard show a different "Net Revenue" than the Finance report? Since metrics live in silos, it is difficult to govern them and ensure they are discoverable and trustworthy for reusability throughout their lifecycle.

In this post, we will share how Zalando is achieving this by leveraging the full breadth of the Databricks Platform. We will dive into how we are building a Unified Semantic Layer that bridges the gap between Transactional Data and Analytical Data. Specifically, we will cover:

To manage our vast data landscape effectively, we decided to move away from resource-centric gatekeeping. In that model, every new dataset or consumer required bespoke IAM roles, S3 bucket policies, and exception handling. But we identified challenges: permissions were fragmented across thousands of resources, cumbersome to review, and prone to drift. Therefore, we shifted to an identity-based governance approach. Access decisions are expressed as reusable policies tied to people and groups. They are evaluated consistently across datasets and enforced centrally. This makes access easier to operate, audit, and evolve as teams and data change. We built this foundation using Databricks Unity Catalog and implemented a federated access control framework on top.

We designed a dual-catalog pattern that strictly separates the creation of data from its consumption, ensuring that agility doesn't come at the cost of control: