Changelog
Back to All

Empower Data Platform v1.36

3 months ago by [email protected]

👍

Key Changes & Improvements

Note that due to the holidays, work for versions 1.35 and 1.36 has been combined into a single release: 1.36.

  • PowerBI Support for Publishing Data Flows: Empower now allows you to refresh a PowerBI dataset directly from the UI when adding a Publish step using Workflows and Data Flows.
  • Menu & Naming Convention Enhancements: Key changes have been made to naming conventions and menu navigation for improved intuitiveness, including a direct link to documentation and an improved menu collapse button.
  • Workflows Enhancements: A new "Runs" tab has been added to the Workflows Overview screen, providing a complete list of all runs along with detailed information about each workflow step. We've also added filtering and tooltips to make navigation use even easier.
  • Cornerstone Self-Service Connector: You can now setup Cornerstone connections from the Empower UI.
  • AWS S3 Bucket Self-Service Connector: You can now setup AWS S3 Bucket connections from the Empower UI.
  • SFTP Connector Enhancement - Support for CSV and JSON files: The SFTP connector now supports the ingestion of both CSV and JSON files.
  • Databricks Connector Enhancement - New Auth Type Option: A new authentication option using OAuth with a service principal has been added for Azure Databricks access.

📘

PowerBI Support for Publishing Data Flows

When creating a new Publishing Data Flow, you will now have the option to Refresh a PowerBI. This will allow you to automatically schedule a refresh of a Microsoft PowerBI Dataset easily and directly from the Empower platform.

As a reminder, these Data Flow steps can then be added to a Workflow to create a full end-to-end pipeline that will grab your updated source data and automatically refresh your PowerBI reports!

For details on how to use this feature, read more in the Docs here.

📘

Menu & Naming Convention Enhancements

We have made several key changes to our naming conventions to better align with standard terminology, making the platform more intuitive. Notable changes include:

  • "Data Sources" → "Connect"
  • "Data Acquisition" → "Acquire"
  • "Analytics Engineering" → "Transform"
  • "Data Publishing" → "Publish"

These updates are most prominent in our Menu navigation but can be found throughout the platform. Additionally, the menu has been enhanced with a direct link to our documentation and an improved collapse button for easier use.

📘

Workflow Enhancements

The Workflows overview screen has been significantly enhanced with the addition of a new "Runs" tab and improved usability features. The "Runs" tab offers a comprehensive list of all Workflow runs along with key information for each run, and clicking on a record reveals detailed insights into each workflow step.

Additionally, column filtering and tooltips have been added to make navigation faster and more intuitive, allowing you to efficiently find and understand the information you need.

For details, read more in the Docs here.

📘

Cornerstone Self-Service Connector

Cornerstone is now available as a self-service connector. Setup connections to Cornerstone using the Empower UI.

For details, read more in the Docs here.

📘

AWS S3 Bucket Self-Service Connector

AWS S3 is now available as a self-service connector. Setup connections to a AWS S3 Bucket using the Empower UI.

For details, read more in the Docs here.

📘

SFTP Connector Enhancement - Support for CSV and JSON files

The SFTP connector has been enhanced to support the ingestion of both CSV and JSON files, expanding its versatility and allowing for more flexible data integration.

For details, read more in the Docs here.

📘

Databricks Connector Enhancement - New Auth Type Option

A new authentication option using OAuth with a service principal has been added to the Databricks connector, providing a more secure and streamlined method for accessing Azure Databricks.

For details, read more in the Docs here.

Itemized Changelog

Enhancements & New Features

  • Reengineered Watermark functionality for Data API.
  • Reengineered publish process for SQL Server and Azure SQL DB in Publish Connector.
  • Made Interactive Clusters work for Spark and custom connector data sources in Data Factory.
  • Changed Databricks authentication type for improved security.
  • Created workflows runs overview tab for better monitoring.
  • Renamed menu items for empower navigation.
  • Added documentation menu item to the sidebar.
  • Implemented workflow name filtering support.
  • Created a Cornerstone connector.
  • Extended SFTP connector to support different file formats.
  • Added SQL Server & Azure SQL DB CDC support.
  • Created JDBC Connector for Azure MariaDB.
  • Developed S3 Bucket Connector.
  • Created a PowerBI refresh flow type.
  • Updated publishing page to display PowerBI flows.
  • Added instructions information modal on PowerBI flows.
  • Added tooltips for workflows overview tab.
  • Added support for migrating flows of type 'CUSTOM' in Config Migration.
  • Updated Metadata Extraction Process to update Last Connected datetime for Data API.
  • Rewrote code for Azure SQL DB JDBC Connector.
  • Enhanced PowerBI workflow runs to save 'workspaceId' and 'datasetId'.
  • Created a script to fix legacy MergeStrategy setup in StateDB.
  • Refactored workflows pages to utilize shared state.
  • Updated create new flow on the publishing page in UI for PowerBI Flows.
  • Ensured PowerBI Flows appear in workflows 'Publishing' tab.
  • Updated flow run history for PowerBI flows to surface a details pop-out blade.
  • Removed 'View' button from System Versions page in empower UI.
  • Clicking workflow name from Workflows overview now routes to /history instead of /config.
  • Updated menu to remove blue collapse arrow on Settings sidebar.

Bugfixes

  • Fixed issue where Interactive cluster incorrectly runs when passing -1.
  • Resolved candidate key overwrite issue.
  • Corrected error message display when user clicks multiple times on Complete Migration button during Data Migration.
  • Fixed inability to schedule runs for a future time on the same day as the start date in Workflows.
  • Addressed Max Watermark Logic issue in Extract Stored Procedures causing problems with Synapse Link Loading.
  • Resolved issue where POST /promotions/sources/ can't create data source on the target environment in Config Promo.
  • Fixed SQL Server error preventing user from creating Flows migration preview in Config Promo.
  • Corrected Pipeline behavior to properly fail records in case of AZURE_QUOTA_EXCEEDED_EXCEPTION (CLIENT_ERROR) for PL_PUBLISH_DATA.
  • Addressed queueing issue for single objects for ingestion in MariaDB and MYSQL.
  • Ensured Custom Flows work by resolving pipeline failure.
  • Fixed issue where DAQ Flows ingestion filter dropdown does not show "Queued" state filter option.
  • Made three dots menu visible for the last records on the grid in Workflows.
  • Ensured workflow name updates correctly in the workflows configuration page.
  • Aligned Flow Run Logs Grid and traffic light component in DAQ.
  • Resolved filter icon issue in Chrome where it is shifted and not fully visible when sorting Data Publishing Flow Run Logs.
  • Fixed issue where Runs tab does not show data on opening the tab for first time after login in Workflows.
  • Ensured data loads on page refresh for Workflow Runs.
  • Fixed Save & Test failure for sources of type MariaDB_New and Postgres_New.
  • Ensured workflow history loads when switching from Configuration page to Runs in Workflows.
  • Corrected workflow status change when running workflow from Configuration page.
  • Adjusted delete workflow confirmation modal to display as expected.
  • Updated restore workflow confirmation modal message to be as expected.
  • Corrected delete confirmation message on an Archived workflow.
  • Fixed flows order at the Run/Schedule modal when opened from Overview page in Workflows.
  • Ensured Workflow Name is displayed for all flows.
  • Updated "Last Modified by" date and time after updating name and description in Workflows.
  • Fixed issue where switching between Configurations to Runs tabs, updated flows are not getting saved in Workflows.
  • Resolved access issue where users are getting logged off.
  • Displayed running state in "Info" column in Workflows.
  • Ensured Workflow column is sorted alphabetically without case sensitivity.
  • Fixed deselection issue of filter values unrelated to the first column selection in Workflows.
  • Corrected navigation from specific Run back to Overview page in Workflow Runs.
  • Adjusted confirmation modal text to correctly show Archive, Restore, and Delete Data Source in Connect.
  • Updated naming inconsistencies where Acquire flow shows as Data source while creating new Acquisition flow.
  • Corrected toast messages to show as Data Source for Archive, Restore, and Delete in Connect.
  • Fixed modal text showing Deleting an Acquire flow as Data Flow in Acquire.
  • Corrected flow status text showing as Data Publishing for new Publish flows.
  • Ensured "Last Modified by" date and time are updated after toggling Schedule Active state from configuration page in Workflows.
  • Updated Icons for Acquire and Transform on the Workflow - Add flow screen.
  • Fixed editing transformation name display issue in Transform.
  • Ensured flow run ID is copied correctly to the clipboard from the Flow Run Details modal in Workflows.
  • Corrected various name updates across the UI.
  • Fixed Delete, Archive, Restore functionality for Acquire and Transform flows.
  • Resolved the wrong Flow Group names being returned from the API in Data Migration.
  • Updated menu to remove blue collapse arrow.
  • Ensured PowerBI Flows appear correctly in workflows 'Publishing' tab.
  • Updated pop-out blade for detailed flow run history for PowerBI flows does not surface.
  • Removed 'View' button from System Versions page.
  • Corrected routing behavior when clicking workflow name from Workflows overview to /history instead of /config.
  • Fixed display issue with Workflow Name column title at the "Runs" tab.
  • Resolved error in the console when switching tabs in System Settings.
  • Fixed "All Other Configurations" toggle name at the New Migration screen in Data Migration.
  • Corrected element placement issue in the sidebar under Settings.
  • Fixed remaining naming inconsistencies across the UI.
  • Ensured workflow name is displayed when restoring workflow in Workflows.
  • Corrected behavior to prevent search on Active/Info/Schedule/Last Runs in Workflows.
  • Ensured ability to remove the workflow in configuration in Workflows.
  • Updated toast message to show correct flow name when creating new Refresh PowerBI flow.
  • Ensured all filter dropdowns show appropriate options when the grid is empty in Workflows- Overview.
  • Fixed issue where the add flow modal doesn't open right away on click.
  • Ensured last item in Configurations is fully visible in Workflows.
  • Corrected "Source Deleted" chip text to "Connection Deleted".
  • Ensured Publish flow appears correctly as Publish in Workflow history page.
  • Updated toast message to correctly indicate Workflow is saved.
  • Fixed duration update for running workflow state.
  • Ensured flow status is updated on running without refresh in Publish.
  • Fixed spacing issue between last two items at skeleton loader in Sidebar.
  • Corrected Cancel button functionality on Workflow schedule page.
  • Ensured "Archived + Scheduled" Workflows are not displayed if "Not Scheduled" filter option is selected.
  • Fixed issue where Archive workflows should not show previous schedule in Schedule column in Workflows.