👍

Key Changes & Improvements

• Batch and Cluster Configuration: Users with advanced options enabled can now choose to enable batch extraction and configure clusters for Data Flows.
• Transform Flows - Skipped Status Support: Improved the Transform run history logs by clearly distinguishing entities that are skipped due to a predecessor's failure to enhance root cause identification and debugging efficiency.
• Connection Schema Name Validation: Added validation to entering the same name for Bronze and Silver schema inputs when setting up a connection to avoid downstream failures.
• Redshift Self-Service Connector: You can now setup Redshift connections from the Empower UI.
• OneDrive Self-Service Connector: You can now setup OneDrive connections from the Empower UI.
• Connector Support for Excel Files: Support for Excel files (.xls and .xlsx) has been added to the Sharepoint, AWS S3, SFTP connectors.

📘

Batch and Cluster Configuration

Users with advanced options enabled can now choose to run acquisition flows using batch or non-batch extraction.

Additionally, users can configure the Spark cluster by choosing the cluster type, node size, number of workers, and job concurrency. For interactive clusters, users can specify the name of the cluster they want to use.

For more details on how to use these new advanced settings, read more in the Docs here.

📘

Transform Flows - Skipped Status Support

We've updated the Transform run history log to clearly show entities that are skipped due to a parent entity's failure. These entities will now be displayed in gray with a skipped icon (a circle with a line through it). This change helps you quickly identify the root cause of failures. Additionally, you can still click on skipped entities to view any associated messages, ensuring you have all the necessary information for effective troubleshooting.

For more details, read more in the Docs here.

📘

Connection Schema Name Validation

To prevent downstream Data Flow run failures, users will now see an error message if they enter the same name for both the Bronze and Silver schema inputs when setting up a connection. The Save and Test button will be disabled until the names are different.

📘

Redshift Self-Service Connector

Redshift is now available as a self-service connector. Setup connections to Redshift using the Empower UI.

For details, read more in the Docs here.

📘

OneDrive Self-Service Connector

OneDrive is now available as a self-service connector. Setup connections to OneDrive using the Empower UI.

For details, read more in the Docs here.

📘

Connector Support for Excel Files

We've expanded our SharePoint, AWS S3, and SFTP connector capabilities to include support for Excel files (.xls and .xlsx). You can now seamlessly connect to and import data from Excel files stored in these locations, making it easier to integrate Excel data into your Workflows.

For more details on how to use this new feature, read more in the Docs here: Sharepoint, AWS S3, SFTP.

Itemized Changelog

Enhancements & New Features

• Created a Spark connector for SAP Concur integration into Empower Spark.
• Created a Spark connector for Redshift integration into Empower Spark.
• Added connection schema validation for Data Source.
• Implemented created by filter in Workflows.
• Added and displayed filtering for runs modal in Dashboard.
• Enabled sorting for Created By column in Workflows.
• Added sort by Schedule column in Workflows.
• Executed Custom Data Factory Pipeline as part of a Custom Flow.
• Executed Databricks Workflow from Data Factory.
• Created OneDrive Connector.
• Extended connectors (Sharepoint, AWS S3, SFTP) to support XLS, XLSX files.
• Updated flows to support cluster configuration.
• Fixed batch extraction issue with non-spark sources in Data Factory.

Bugfixes

• Fixed issue where user cannot set custom "Repeat On" flow schedule in Empower UI.
• Resolved problem where workflow breadcrumbs are lost after opening settings.
• Corrected issue where user can't select filter option by clicking on the checkbox in Empower UI.
• Fixed Expires parameter having a value on schedule when it shouldn't.
• Corrected dialog box loading issue the first time when trying to schedule flows in Empower UI.
• Resolved publishing failure due to duplicates error at the target environment.
• Fixed issue where the "Refresh" button at the Run Logs blade doesn't update Duration in Flows/Workflows.
• Ensured Access Control search works at any level and doesn't throw errors in console.