Analytics Engineering Entity Graphs: You can now visualize and manipulate entity dependencies for Analytics Engineering flows using our new graph interface. Create new nodes, edit existing connections, and remove obsolete dependencies in a new experience! View the documentation to learn more about dependencies.
New Self-Service AEC Connectors!: View our Source System Support page for details on all of our supported data source types. This release, we've expanded our Architecture-Engineering-Construction source type set with the following connectors:
Metadata Catalog Default Behavior Change: New objects discovered during all new metadata extractions will no longer be enabled by default. This change occurs for all new and existing data source connections you manage in your deployment.
Empower for Dynamics Accelerator Notebooks: Empower now ships with notebooks to assist Empower for Dynamics integrations. See the notebook documentation here.
Public Preview - Direct SQL Staging: We are publicly previewing a new feature in our model building process. You can write a SQL statement directly within a Databricks .SQL file, specify the filename in the Entity's notebook column, and the query will be run during entity buildout.
📘
Analytics Engineering Entity Graphs
View and manipulate entity relationships in the overhauled Entity Dependency experience. View the configuration for any existing AE flow and navigate to the Entity Dependency tab.
Visualize entity relationships and buildout order.
Netsuite Connector is a powerful tool that ensures seamless integration between NetSuite ERP and other business applications.
Origami Risk API serves as a powerful data source, providing businesses with comprehensive solutions for risk management, claims administration, and policy management.
Self-Service Capable - create and configure connections to these new sources entirely within the UI.
Documentation Available - learn how to configure and maintain these connections on their respective documentation pages.
📘
Empower for Dynamics Accelerator Notebook
Empower for Dynamics, the built-in tooling that supports Lakehouse-to-Dynamics integrations and migrations, now comes with a notebook that accelerates your development. View these notebooks in the shared utilities file within your Databricks workspace, and learn more reading the notebook docs.
📘
Public Preview - Direct SQL Staging
An alternative to using a staging notebook for a model entity, you can write a SQL statement directly within a Databricks .SQL file, specify the filename in the Entity's notebook column, and the query will be run during entity buildout.
Simple entitles built this way can be up to 60-70% faster than a staging notebook, and model builder will automatically handle the boilerplate wrapper code for you. Faster runtimes and development times!
We are actively building out documentation for this as it moves to general availability in v1.30. Please contact your delivery team for more details about this previewed feature.
Itemized Changelog
Enhancements & New Features
A new connector has been created for integrating Origami into Empower Spark, enhancing data ingestion capabilities.
A new connector for NetSuite has been developed, improving data extraction and loading processes.
Enhanced DDU to support the insertion of inactive rows, improving data management flexibility.
Implemented incremental load with filter for SAP SAC.
Metadata Extraction Default Behavior Update: New objects now default to off for new and existing data source connections.
A new template notebook has been introduced for Empower for Dynamics, simplifying the development process.
New table modes supported in ingestion: RAW DATA TABLE and HISTORY TABLE. Note: not UI-accessible yet.
Enhanced the API to include previous promotion blob path in PromotionTable.
New models for advanced filtering on target environment and migration status in the Data Migration have been created.
Enhanced filtering on Log - Ingest and Log - Extract pages to support querying with Database ID List, allow users to find logs faster.
Updated APIs to support an array of Database IDs.
Removed the google-libphonenumber node module from empower-ui.
New Entity Dependency creation, entity connection UI, and enablement of connection and node deletion in the entity dependency view have been implemented.
Created a mini map and controls for better navigation and visibility within the Analytics Engineering Graph View.
Public Preview - Direct SQL Statements are now public-preview supported during model buildout using the entity's "notebook" column to specify the path of the SQL file.
Bugfixes
Connector Bugfixes: Various issues related to ingest and extract failures across platform connectors like Workday RaaS, SAP SAC, Origami, and Netsuite have been resolved.
WEB PUT FAILURE Error Handling: Improved error handling for WEB PUT FAILURE steps not catching errors from ADB in SPARK_PL.
UI Enhancements and Fixes: Multiple UI issues related to AE Entity Dependency, Metadata Catalog, and advanced options in the ingest logs have been addressed, improving user interaction and visual presentation.
