Empower Data Platform Version 1.6

👍

New and Enhanced Source Support

With this release we added ServiceNow support, and enhanced the robustness of the Dataverse and MariaDB sources.

📘

Preparation for Enhanced Spark Packages

We are rolling our more infrastructure to be able to perform advanced data movement in the Spark layer for customers. We expect substantial savings to stem from this move -- as evidenced by our preview customers.

Data Acquisition

1. Added support to extract ServiceNow data using Spark.
2. The Dataverse extraction is now able to process corrupted D365 Migration files.
3. Updated MariaDB metadata extraction to cast set and long text types to the appropriate Empower data types.

Configuration & CI/CD

1. Added support for multiple contexts to the CLI so that a user can execute commands related to different virtual environments. i.e., configuration promotion.
2. Added email notifications for Hitachi employees when a customer's Databricks notebook deployment fails.
3. Decoupled the Data Factory deployment process to allow more flexible and rapid development.

Databricks Integration

1. Added a script and installation process for JDBC drivers on Databricks Clusters.
2. Improved how the Empower Core Spark package handles imports.
3. New libraries come preinstalled on the empower data engineering Databricks cluster. These are "google-analytics-data" and "azure-storage-file-datalake."

Security

1. Added Azure Key Vault logging support to monitor security audit logs from Empower key vaults.
2. Added delete locks to the customer's datalake to improve data security.
3. The main Azure Data Factory pipeline now calls the Empower API for an authorization token.
4. Automate supplying authentication parameters to the Data Factory.