Changelog
Back to All

Empower Data Platform v1.24

9 months ago by [email protected]

👍

Empower UX capability and Data Platform updates

  • Data Source UI Improvements : Metadata Catalogs are now viewable directly from the Data Source page. Quickly configure global acquisition settings with fewer clicks!
  • Change Feed Ingestion Support: Empower's ingestion engine can now support Change Data style merges. Note, this feature is still in private preview!
  • Configuration Migration Improvements: You can now migrate archived data source credentials too.
  • New Data Connectors: Azure DevOps is now available from the UI. Connect, explore, and unleash your data's full potential!

📘

Data Source UI Improvements

Metadata Catalogs are now viewable directly from the Data Source page, in addition to the Data Acquisition Page. As we start to rollout Data Flow's in the upcoming releases, our modifications to Empower's UI will enhance the user experience and provide greater self-service capability.

Additionally, you can now define custom bronze and silver schema names for Data Sources in the connection configuration pane. Note: you must define these fields before your first ingestion of data, as the system will ignore changes to this field post-initial-acquisition. See more details in the Data Sources user guide.

📘

Change Feed Ingestion Support - Private Preview!

By customer demand, we have built out Change Feed merge support within Empower's ingestion engine. In this merge strategy, duplicates are inserted into the Bronze Layer as non-current records. Every incoming piece of data is treated as a change to a record.

If your source contains deletion metadata, Change Feed can support it even for incremental extraction! This is not a feature we yet support for most deployments. Please work with your delivery team for enabling this for your deployment.

📘

Configuration Migration Improvements

In addition to existing functionality today, you can now migrate archived data sources as well as restored ones from the Data Migration pane. Note, archive restrictions still apply with these migrated sources: you'll need to restore them for editing.

📘

New Data Sources Available

New sources ready for self-service configuration for this version include Azure DevOps. All data source drivers have been updated where applicable for better handling in general.

Itemized Changelog

Enhancements & New Features

  • Updated all Source drivers and performed sanity checks.
  • Added a Source File Column in Synapse Link connectors.
  • Supported Change Data Capture in Ingestion Engine.
  • Improved Data Sources and Acquisition with Metadata/Data Extraction modifications.
  • Created an Azure DevOps Connector.
  • Updated metadata extraction status to support new successful status state.
  • Removed old implementation of expression evaluation lexical parser.
  • Included support for Acquisition Flows.
  • Added optional text entry field Item Name to Execute.
  • Updated /data-sources page with new 'View Metadata Catalog' button.
  • Created new pages and implemented AG Grid on metadata-catalog page.
  • Displayed Duplicate Keys Error and added functionality to data acquisition search component.
  • Added edit and delete acquisition flow capabilities.
  • Displayed archived data sources as new migration options and updated what to send to API when migrating all assets

Bug Fixes

  • Resolved an issue where no pg_hba.conf entry for host caused a fatal error.
  • Fixed a Dynamics Publishing failure bug.
  • Solved a variety of 400-level errors.
  • Fixed failed execution of stored procedure state.InsertMetadata.
  • Resolved errors during ingest about being unable to create delta tables.
  • Addressed issue where Native ADF style connectors fail to connect if fields contain semicolons.
  • Fixed access denied error when trying to get the latest metadata from Azure DevOps.
  • Resolved missing fields for Azure DevOps in FrontDoor Service.
  • Fixed the issue where users were unable to cancel the new source form.
  • Corrected invalid request error when closing an empty modal for New Record.
  • Resolved inactive 'View Metadata Catalog' when opening source detailed view.
  • Fixed issue where Run/Schedule modal contained objects disabled on a global level.
  • Addressed the problem where archiving/restoring of data source does not update button at the data source detailed view page.
  • Fixed incorrect optional text display on Data Sources.
  • Resolved UI issue where changes to Source Configuration were not shown.
  • Fixed animation triggered when a user with the wrong permission tries to create a new connection.
  • Corrected source type name not fully displayed at the Data Source dropdown.
  • Fixed wrong behavior on navigation back from the history page for Data Acquisition flow.
  • Addressed issue where migration details were not fully visible in case of a long assets list.
  • Fixed the problem where applied filters count was not displayed when Estimated Rows filtering applied with 0 as a min value.
  • Resolved issue where the search suggestion dropdown scrolled with the page.

Other Notes

We've added a few pages to our public doc site:

  • Our new Product Principles page defines our North Star for feature development.
  • The Empower lexicon is now contained within our new Core Concepts page; check it out to educate yourself on the Lakehouse-integration vision!