Trying to understand shared-integration and integration

[baatch]

Hi !

I have read the documentation several times and tried to wrap my head around the concepts of the Enterprise Scale.

I don't quite understand where to put the different resources and what the difference is for shared and non shared integrations according to the reference architecture. There are data factories for ingestion and for integration, what is the difference.

So lets ju give a scenario that I have and maybe you could give me some guidance on what is integration, ingetsion, data product and where to put the resources?

[marvinbuss]

Thanks for the feedback @baatch! Let me try to make this clear:

1. Data Factory in runtimes-rg
   The Data Factory hosted in the runtimes-rg is just acting as the initial connection point for the Self-hosted Integration Runtimes (SHIR) inside the Data Landing Zones. The SHIRs just connect to the Data Factory, but no actual workloads are executed here. This is just for managing the SHIR and for troubleshooting purposes of the SHIR instances. This separation also allows us to follow the least privilege principle and allows us to provide elevated access to a user, if a SHIR is not running as expected or a node is down. From this Data Factory instance, the SHIR can be shared with other Data Factory instances, so that they can also make use of the runtime. Sharing these instances reduces cost, management overhead and simplifies development work if not each and every integration team needs to be concerned with hosting a SHIR. Often, the IPs of the SHIR also need to be whitelisted in the central NVAs or network appliances to successfully access data sources from on-premises, which is another reason why it makes sense to share the SHIRs in the Data Landing Zone. This way, the teams do not have to wait for the IP whitelisting each time they want to access data sources outside of Azure.

2. Data Factory and Databricks in shared-integration-rg
   The Data Factory in shared-integration-rg is mainly used for the metadata driven ingestion framework that many organizations have setup today. For this, it also uses the shared SHIR from 1 which is automatically setup by the IaC that we provide. If such a framework does not exist, organizations can also leverage the Microsoft build metadata driven copy activity. The ADF uses the SQL Server and a Key Vault in the metadata-rg to store the metadata and secrets for this ingestion framework. The Databricks instance in the shared-integration-rg is used for the data engineering of these raw datasets, to clean the datasets, apply a common data file format and make other necessary adjustments.

3. Data Integration RGs
   If Data Source owners can and want to own and develop the ingestion process of data assets into the data landing zone on their own and do not want to rely on the metadata driven framework, they can also develop and implement this on their own. For this scenario, we have the data-integration-rg's. This will most likely also be used for any streaming data assets that should land into the platform. For these patterns, you can essentially also use either of these IaC templates to get started with the custom integration development work:

• https://github.com/Azure/data-product-batch
• https://github.com/Azure/data-product-streaming
  We are also stating in the prerequisites document that these templates can be used both for product and integration development work. The main difference will most likely be that integration teams also get access to the SHIR in order to load data assets outside of Enterprise-Scale Analytics (ESA). In the streaming use case, IoT Hubs or Event Hubs will be used to ingest the data instead of a SHIR.

Please let me know if you have more questions.

[marvinbuss]

Hi @baatch.
Please see below:

1. Storage Accounts
First of all, the "external" storage accounts are available for push patterns. Several organizations have tools that require access to a storage account key to push data to a Data Landing Zone. For these scenarios, data should be first landed on an external storage account, as the key would give global access rights to a storage account/data lake. No one should have global access to a data lake as they would gain the power to alter or delete any dataset on the datalake. Hence, we are blocking key authentication for the raw, enriched&curated and workspace data lakes.

Since ADF, EventHub and an Azure Function can land data using a MSI or SP, you can land data directly in their respective format in the raw datalake. The enriched&curated versions (common data format + potentially some cleansing applied + aggregation) can then land in the second data lake. From here, these datasets can be shared with other Data Product teams to build ML models, enrich the datasets further, combine it with other business information etc.

2. Databricks
Databricks is always a shared resource within a Data Landing Zone. This is due to several reasons:

• No great ARM and Azure Policy integration
• Policies can only be defined within a workspace on the data plane to control cluster configuration and enforce connections to external Hive metastores, logging and other settings.
• Otherwise, the Data Platform team cannot make sure that the external metastore is actually used and configured. If not configured, certain users may not be able to access the data via SQL Table.
• Some of the setup also requires access to Keys (SQL Server Keys) which should definitely not be handed over, as this again creates a potential risk.
• New subnets are required each time a new Databricks workspace gets deployed. Data Product teams only get access to their own dedicated subnet on the shared vnet and would have to request access to additional subnets if they want their own Databricks instance.
• Databricks user provisioning should happen via AAD SCIM synch to make sure that users leaving the organization or business unit, also loose access on teh Databricks side.
  All these steps are required from a management perspective. Hence, Databricks creates some management overhead from a service configuration and service management perspective. Due to this we have only shared Databricks workspaces that can be used by Data Integration and Data Product teams.

3. Shared Integration RG vs. Independent Data Integration RG
May I ask one question before I provide a definitive answer? Is the team planning to move this towards a more metadata driven solution in the near future for this whole solution or are they planning to continue leading the development work in a more "manual" way?

[baatch]

@marvinbuss thanks again!

See below:

1. Storage Accounts
   Ok then I can change the storage from external to Raw instead no problem.

2. Databricks
   Great explanation about Databricks and why it always shared in this design. I have seen other architectures (without private link) that would provision Databricks workspaces for every data product.

3. Shared Integration RG vs. Independent Data Integration RG
   The team have plans to build a metadata driven solution in the near future. Would you care to explain the difference and how one should think of when to use Data Factory or Event Hubs in shared vs independent ? Would you mind mapping placing the resources in the picture in their correct Resource Groups?

[marvinbuss]

1. Storage
   Perfect! Sounds good to me!

2. Databricks
   We have decided against a Databricks workspace per Data Product, because of the reasons mentioned above. Metadata synch is another issue that I did not mention above. If one Data Product teams wants to consume data via SQL (Hive metastore table definition), you can either make all the teams use the same External Hive metastore or you have one per team. If you have one per team, it will become difficult to share the metadata between workspaces and synching Hive Metastores is also not trivial. If you use a single consolidated one, you may run into conflicts, if two teams try to write at the same time. In summary, we have decided to use centralized shared Azure Databricks instances.

3. Shared Integration RG vs. Independent Data Integration RG
   If the team is planning to refactor the existing solution to a metadata driven solution, I would propose to use the shared-integration-rg in combination with the Key Vault and SQL Server in the metadata-rg for keeping the metadata.
   The shared integration RG should be used for the metadata driven framework that many have build, to provide a quick way of onboarding datasets. However, these integration patterns are usually quite standardized and there is sometimes no clear owner assigned to the respective datasets who is taking care of data quality, versioning and enhancing of the dataset based on feedback.
   To allow for more custom integration patterns, we have the concept of Data Integration teams (data integration RGs). These teams own their RG as environment and own the the end-to-end responsibility from ingestion to the serving layer for data domains or sub-domains. This will lead to a more distributed data ownership and data management model, where, similar to data products, teh data integration responsibility is also distributed across various smaller teams, who own the integrated dataset as a product. The Data Integration teams own a domain or sub-domain capability and must be encouraged to serve datasets for access by any team for any purpose or project they may be working on. These integration teams also allow for a quicker feedback loop between data provider and data consumer which results in much shorter and quicker prioritization, development cycles and a more agile development process. In an ideal world, the data integration would be owned by the same team that owns the source systems. But the team should in general not only consist of data engineers that work on the source systems, but also of subject matter experts (SMEs) for these datasets, cloud engineers and data product owners. Such a cross-functional team setup reduces the communication required to teams outside and will be essential when developing the complete stack from infrastructure to actual data pipelines.

We will soon release an article to make scaling and self-service as well as the ownership model a bit more clear. Stay tuned. I hope this is helpful!

[baatch]

2. Databricks Unity Catalouge :) ?

3. Great ! I think I read some of updates in the doc guidance updates in your PR that helps alot with the understanding (https://github.com/Azure/data-management-zone/pull/204/files?authenticity_token=pPMlE14iKfaKCi8Kl3TI6MoyXN%2F9Q8gooO6OGuiAmXjDdmGayttWBT8iuwpQj2DULVvsGYbPYaLj3Z6NYzdgYw%3D%3D&file-filters%5B%5D=.md#diff-87f77847dff4ae08b544d966525c85b94e8732744f42b5a701c21ee022e03e45)

[marvinbuss]

The Unity Catalog may solve the metadata issue, but none of the other issues. :)
If we see some synching mechanism between Purview and Databricks, that may be an option.

Yes, the doc will be published to CAF sometime soon. I am waiting for a review from some colleagues, before we publish this in CAF.

Thanks for the great discussion!