👍

New Features in v1.26!

• Data Flows: Finally in public preview, Data Flows allow you to self-service your data management in an intuitive way!
• Advanced Options Upgrades: Renaming tables, powerful filter capabilities, and a general column cleanup to reduce clutter.
• About Page: From a brand new page found in Settings, you can now view (and copy!) the version of every service supporting your environment.
• SAP SAC Connector: The SAP SAC connector for data acquisition is now available for self-service use.

📘

Data Flows - Public Preview

Introducing the third-generation of metadata-driven pipeline orchestration, Data Flows! Data Flows simplify the complex parts data movement by breaking it down into modular parts. Data Flows come in three types:

1. Data Acquisition: metadata extraction, data extraction, and ingestion into the Lakehouse from supported data sources.
2. Analytics Engineering: DAG-style notebook execution & model building.
3. Data Publishing: pushing data out of the Lakehouse to supported data sources.

Configure Data Flows, trigger them to run on-demand, and schedule them to run at a regular occurrence within the Empower UI.

As a public preview feature set, we do not recommend using Data Flows for production-level pipelines just yet. However, do feel free to use them in your development processes and please let us know your feedback! For production-level pipelines, continue to use Empower's Step Command system.

Read more about Data Flows in our brand new user guides.

📘

Advanced Options Upgrades

We've renamed many of the tabs within Advanced Options to be more intuitive to our users. DB2Step, for example, is now named Orchestration, since its where users can configure ELT through Empower's 2nd generation Database to Step Command system.

Model-related tables have been renamed (Entity -> Model - Entity) for clarity. Log tables have been also renamed to designate their purpose (Log - Model, e.g.).

Finally, may tables now contain useful filtering capabilities. You can now filter by Model on the Model - Entity and Model - Entity Column tables, as an example. See the itemized changelog below for the full review of all Advanced Options updates.

As there are several overlapping pieces of functionality between Advanced Options and Data Flows, we will be keeping the Advanced Options documentation around, while we continue to build out new documentation for Data Flows as the feature set matures.

📘

About Page - Empower Services Versions

Visit the new About Page from the Settings module to view the live version of every Empower service powering deployment. Check on your deployment versions. Verify the completion of a new version upgrade. Easily copy out versions and send to your admin or Hitachi delivery team during integration debugging.

Today, the About page provides version information for the following services:

• UI
• Core API
• Data API
• Empower API (depreciated)
• Empower Spark
• Databricks
• Datafactory
• Infrastructure

Learn more by reading about the Settings module and the About page.

📘

SAP SAC Connector Available

SAP SAC is now a supported self-service connector for data acquisition in Empower. View our global collection of suppported sources in our docs, and view our revamped data connector documentation for each connector, including the SAP SAC page.

🚧

Other Notable Enhancements and Fixes

• Fixed a Google Analytics bug that prevented successful ingestions.
• Addresses a variety of 4XX and 5XX level errors granting much greater platform stability.
• Built a template data modeling staging notebook and user guide, found here.

Itemized Changelog

Enhancements & New Features

• Public Preview release of Data Acquisition, Analytics Engineering, and Publishing Flows and respective UI modules.
• New About tab in the System Settings page, containing all Empower Service versions.
• Updated config promo source DB with the latest data.
• Included INTERNAL_VERSIONS_URL in the CoreAPI Container.
• Built a template stage notebook and user guide.
• Supported Business Effective Dates for Type 2 Dimensions.
• Modified Publishing Flow Pipeline to improve efficiency.
• Widened Empower DDU Logging Columns for better readability.
• Updated extract and ingest methods to support optional filtering.
• Modified Empower Spark's Ingestion engine to support flow execution.
• Support for friendly system reporting display names.
• Bulk update include toggle support for Data Acquisition flow configurations.
• Advanced Options specific improvements:
  • DB2Step renamed to Orchestration.
  • Flow Config renamed to Execution/Scheduling.
  • Entity renamed to Model - Entity.
  • Entity Column renamed to Model - Entity Column.
  • Entity Dependency renamed to Model - Entity Dependency.
  • Publish Entity renamed to Publish - Entity.
  • Object Model renamed to Log - Model.
  • Object Ingest renamed to Log - Ingest.
  • Object Extract renamed to Log- Extract.
  • Publish Execution Log renamed to Log - Publish.
  • Filtering on Success/Failure on all Log - tables.
  • Filtering by Data Source on Log - Ingest and Log - Extract tables.
  • Unused column cleanup in Orchestration table.

Bugfixes

• Fixed issues causing failed responses from CoreAPI and System Versions endpoints.
• Addressed permissions and missing notebooks issues that were causing failures in Extract & Load.
• Resolved issues with metadata acquisition and ingestion process errors.
• Fixed extra columns issue in Google Analytics causing ingestion failure.
• Addressed various bugs leading to 500 errors, incorrect redirects, and UI discrepancies in Data API and UI.
• Corrected issues with column resizing, breadcrumbs, tooltips, and button naming in Advanced Options & Flows.