Dynamics FO & CE Publishing: These two sources are now also available as data targets for publishing data for both Publishing Flows and in Advanced Options with Load Groups.
Metadata Catalog Object Options: Inside a source's Metadata Catalog, a new column is available that allows key-value pairs to define custom acquisition configuration on an object-by-object basis.
Model Building Improvements: Missing values and ghost records are automatically handled for you within Model - Entity Column by default upon Model - Entity buildout. Direct SQL also now supports incremental entity buildout. See below for more details.
📘
Dynamics FO and CE Publishing
Dynamics 365 FO (Direct) and Dynamics 365 CE (Direct) are now available in the UI to select as publishing targets for both Data Flows and Load Groups. Configure your publishing entities with ease and ship to the Dynamics target of your choice.
You can now specify Dynamics CE - Direct and FO - Direct sources as publishing targets.
A new Options column is available in the Metadata Catalog. Options allow users to define a key-value pairing of special object-specific options for acquisition configurations.
The new Options column available in the Metadata Catalog.
the key-value options modal, visible upon clicking View Options above.
See our user guide on supported key-value pairs from the Metadata Catalogs page "Options" section.
📘
Model Building Improvements
Missing Value Automation: You can now use the missing_value field for entity columns to define a ghost record for when values are missing during entity buildout. To learn more about ghost records and _missing_value_, refer to our documentation on either Model - Entity Column or Analytics Engineering flows.
Direct SQL - Incremental Build Support: Direct SQL was made available for Staging Data Transformations in v1.29. v1.30 enables incremental build support for Direct SQL entity buildout. This is an automatic feature improvement, and no action is needed from users to benefit!
Itemized Changelog
Enhancements & New Features
Model building now allows watermarking when doing an incremental model with direct SQL queries.
Model building now supports missing value -> ghost record automation for more robust entity buildout when columns are missing values.
SQL Spark connector build for Spark extraction to support non-ADF streaming. NOTE: this is currently only available by request. Talk to your delivery team to learn more.
Improved change tracking for MSSQL by modifying candidate key fetch logic.
The Metadata Catalog now contains an Options column to allow for object-specific options.
Dynamics F&O and CE are now available as publishing targets in the UI.
Various UI updates including support for editing missing value columns and null values in source notebooks.
Improved minimap in AE Entity Dependency for better navigation.
Added measures to prevent 404 errors when switching environments.
Bugfixes
Fixed a 500 error when downloading changelog due to permission issues.
Resolved an issue where the connection was closed unexpectedly during Spark publish operations.
Fixed syntax errors in Modeling and Parsing operations, enhancing stability.
Addressed issues leading to data duplication and errors during object ingestion without Flow ID.
Corrected empty Flow tables in StateDB and modeling failures due to missing values.
Improved error messaging by filling out the Job Link field when a notebook fails.
Fixed various UI bugs affecting the AE Entity Dependency graph, records jumping in modals, buggy mini maps, and inconsistent search field behavior among others.
Corrected timezone application to accurately reflect UTC settings.
