Solution pattern · reference architecture
Data governance and lineage as a dependable baseline.
This pattern creates a bounded and practical governance framework for Azure and Databricks data. Owners, terms, access roles, classification, and lineage are connected where daily operations need them.
Starting point
Data exists, but ownership, meaning, and lineage remain unclear when questions arise.
Platforms, reports, and files grow independently. Access accumulates over time, business terms differ, and audits require a manual reconstruction from source to consumption.
Business value
A shared governance core makes accountability and access manageable.
Business owners, technical stewards, and users gain a common view of selected data products. Access and classification become reviewable without turning governance into a documentation-only project.
What becomes measurable
- Coverage of data objects with an owner, classification, and business definition
- Open access reviews, scan failures, and unresolved lineage gaps
- Catalogue usage, metadata freshness, and completion of governance tasks
Reference architecture
Components and their purpose
Purview maps the broader data landscape while Unity Catalog governs Databricks objects. Microsoft Entra ID connects roles, and audit data makes changes traceable.
- 01
Inventory data sources
Microsoft Purview Data Map scans selected Azure, database, and reporting systems through controlled credentials and network paths.
- 02
Maintain terms and sensitivity
Classifications, business terms, owners, and policies are linked to concrete data objects and approved by the business.
- 03
Govern Databricks centrally
Unity Catalog manages catalogues, schemas, tables, models, and permissions in Azure Databricks through consistent naming and role rules.
- 04
Map roles to identity
Microsoft Entra ID groups represent business and operating roles. Direct individual permissions are treated as documented exceptions.
- 05
Connect lineage
Pipeline and query lineage from Purview and Unity Catalog shows how selected data travels from source to data product and report.
- 06
Operate audit and tasks
Azure Monitor and Log Analytics collect relevant events. Owners address access reviews, scan failures, and classification questions through a defined process.
Technology
Concrete services for implementation
The selection is adapted to existing contracts, regions, security requirements, and the actual scope.
- Microsoft Purview
- Microsoft Entra ID
- Azure Data Lake Storage Gen2
- Azure Databricks
- Unity Catalog
- Azure Monitor
- Log Analytics
First project scope
A pilot needs clear boundaries
The first deployment tests data, integration, and the working process in a limited area. It is not a premature enterprise rollout.
Deliberately included
One business data domain with selected sources and Databricks objects, named owners, a small glossary, a role model, classification, and traceable lineage to a relevant use.
Deliberately excluded
No immediate enterprise catalogue, automatic repair of every permission, complete lineage for undocumented legacy systems, or transfer of business accountability to a tool.
Prerequisites and constraints
Technology does not replace data accountability
Data access, responsibilities, licences, and operations must be clear before implementation. Open points are treated as project risks.
- 01Scanners need permitted credentials, network access, and agreed load windows. Not every source provides the same depth of lineage.
- 02Business owners must decide terms, sensitivity, and acceptable use. Automatic classification is only a proposal.
- 03Purview, Databricks, and logging costs depend on scans, data volume, regions, and retention and must be planned.
- 04Data owners, platform operations, identity management, data protection, information security, and audit need a binding working process.
Germany and the EU
Compliance follows the specific purpose
Catalogues and audit logs can themselves contain personal information about owners, users, and access. GDPR minimisation, role-based access, retention, and deletion therefore also apply to metadata. EU regions and cross-border processing paths must be checked for each service. Logs must not be repurposed for employee monitoring without a defined purpose, and the works council must be involved under the Works Constitution Act where that capability exists. This pattern is not an AI system, but it can provide evidence for data and model governance under the EU AI Act.
Free initial assessment
Which data product is most difficult to explain in terms of lineage or ownership?
The discovery call frames the data domain, roles, sources, sensitivity, and an implementable governance boundary.
The initial consultation and joint use-case discovery are free and non-binding.
Related use cases
Compliance and audit evidence retrieval
Evidence sits in tickets, policies, training lists, approvals, and emails. Currency, completeness, and provenance are difficult to see.
Clinical document and SOP search
SOPs, hygiene policies, and device instructions are spread across repositories. Outdated copies remain in circulation while patient data raises the risk.
Real-estate data hub for operating costs and portfolio reporting
Property, lease, energy, and invoice data use different identifiers. Reports are assembled manually and anomalies surface late.