Setup

A number of pieces need to be in place for the DDU pipelines to run successfully. This section will cover the setup of the connectors and Databricks environment.

Package Deployment

All the tooling needed to run DDU applications can be found within the empowerspark docker image. This includes:

• The empowerspark library
• Spark-excel
• The JDBC Jar library for dynamics CRM and FNO

The image can be installed on a Databricks cluster by adding it to the "Docker" tab under advanced options on the cluster UI. For job workflows, the empower spark job policy will automatically add the image to the cluster. Note that for Empower workflows, the docker image will be installed automatically on clusters.

Authentication

The Empower dynamics publishing package authenticates to the Dynamics environment using an Azure Active Directory Application (Service principal). Note that the details for the Azure application are the same for both F&O and CRM environments. Empower SRE team will be able to create Service Principals in Azure AD.

With the service principals created, the following details are required to authenticate to the environment:

1. The Dynamics ERP (either dynamicscrm for CRM or dynamics365 for FNO)
2. Dynamics 365 URL
3. Azure Tenant ID
4. Azure AD Application ID
5. Azure AD Application Secret

In some cases, multiple azure applications may be desired to push data using multiple users. This is handy to by-pass certain api limits when pushing large quantities of data. If desired, create a key vault secret in the linked key vault with the following format. Note that only additional clients should be specified here. The main client details should be specified in the main keyvault secret, discussed below.

[{"client_id": "client_2", "client_secret": "secret_2"}, {"client_id": "client_3", "client_secret": "secret_3"}, ...]

There are two ways to configure this information to be used by the dynamics publishing library.

1. Empower UI

   In the Empower UI, create a new source of either "Dynamics CE - Direct" or "Dynamics F&O - Direct" source types. Provide the required information. If multiple clients are desired, provide the name of the keyvault secret created above in the optional "Client Details Secret Name" field. Empower will then create a new source for the dynamics environment with a related key vault secret.

2. Manually

NOTE: This is an advanced use-case for utilizing the DDU library outside of Empower orchestration.

Create a keyvault secret in the linked keyvault with the following example format:

{"dynamics_erp": "dynamicscrm", "dynamics_url": "my.dynamics.com", "azure_tenant": "abc-123", "client_id": "my_client_id", "client_secret": "my_client_secret", "extra_clients_secret_name": "ddu-publishing-secrets"}

The "extra_clients_secret_name" field is optional and can be used to specify the name of the keyvault secret containing the extra client details.