👍

Key Changes and Improvements

• Analytics Engineering Monitoring: Analytics Engineering monitoring is now possible from the flow log page, giving an entity-by-entity breakdown of the run status.
• Interactive Cluster Support: Empower now supports interactive cluster configuration in addition to job clusters for certain phases in load group pipelines to support large data movement requirements.
• Microsoft Entra - Self Service Connector: New in the UI, you can now setup connections to Microsoft Entra, a comprehensive identity and access management (IAM) solution.
• System Reporting Page Deprecation: We will be removing the System Reporting page in the UI as part of the v1.32 release as we continue to roll out new monitoring and logging features. Stay tuned for more updates in this space!

📘

Analytics Engineering Flow Monitoring and Logging

Empower now offers enhanced capabilities to monitor your Analytics Engineering Flows!

This enhanced logging feature now offers detailed tracking with entity-by-entity status granularity, making it easier than ever to monitor your Analytics Engineering flow processes. Check out the Empower Docs to dive deeper into utilizing these features.

You can now access Analytics Engineering Monitoring directly from your Empower instance by selecting a specific run from the flow's run history!

Stay tuned as we release additional monitoring and logging capabilities for Data Publishing Flows! For the latest updates, visit our Product Release Roadmap.

📘

Interactive Cluster Support

For particularly large jobs, or time sensitive ones that require low-cluster spin-up times, Empower now supports interactive cluster job execution for Extract, Ingest, and Model phases.

If cluster_ids are provided in the job config API calls within the supported phases, jobs will be executed on those clusters rather than freshly spun up job clusters, resulting in lower spin-up latency and providing more flexibility on cluster sizing.

Read the docs, and talk to your delivery team to learn more about how to use this technical feature.

📘

Microsoft Entra - Self Service Connector

Microsoft Entra is a comprehensive identity and access management (IAM) solution designed to securely manage access to resources across a range of environments, including on-premises, cloud, and hybrid setups.

You can now setup Microsoft Entra connections in the UI entirely by yourself. Visit the connector docs for guides on how to setup and use this connector.

🚧

System Reporting Page Deprecation

As part of our updates to logging and monitoring, we will be deprecating the System Reporting page in the UI. You may notice that page will be no longer available in your environment. We plan to entirely replace this page with a new System Dashboard with vastly improved monitoring and logging capabilities later this year. Stay tuned with our Product Release Roadmap for more information!

Itemized Changelog

Enhancements & New Features

• Implemented ingestion check to avoid ingestion collisions.
• Improved Data Publishing pipeline logging.
• Enhanced workflows in Data Factory Pipelines.
• Introduced deduplication logic for Synapse Link sources.
• Developed a Parquet file Data Source Connector.
• Added BeautifulSoup Library for HTML content removal in strings.
• Enabled incremental publishing in Dynamics package.
• Created a Spark connector for Microsoft Entra integration.
• Supported interactive clusters configuration.
• Enhanced custom flow creation capability.
• Updated stored procedures for model management.
• Removed all mount points from code in Spark applications.
• Automated registration for new organizations and backfilled existing ones.
• Refactored UI for Self-Service Onboarding review and deployment submission.
• Added Role-Based Access Control (RBAC) descriptions to all endpoints.
• Implemented a feature for auto-enabling or disabling functionalities via waffle switch.
• Removed feature flagging for flow promotion.
• Updated the Entity Column Model ID for Advanced Options in Modeling Entity.
• Updated endpoint for detailed flow history.
• Enhanced API to support new migration types in the future.
• Implemented a pop-out blade for detailed flow history.
• Introduced stoplight progress bars for Object Model logs.
• Updated tooltips for modeling status indicators.
• Implemented refresh button for AE Flows flow run details.

Bugfixes

• Resolved errors across multiple widgets due to premise service issues.
• Fixed errors in various load groups and data processing failures.
• Corrected behavior for notebook error handling and user configuration issues.
• Addressed connector linked service changes and metadata failures.
• Ensured proper failure mechanisms for custom flow pipelines and source type errors.
• Fixed UI issues in Account Settings and Self-Service Onboarding pages.
• Corrected behavior for environment setup limitations and search functionalities.
• Resolved display issues in Flows UI when switching between running and completed flows.
• Fixed an issue where updating a flow with an empty "description" field was not possible.
• Addressed incorrect completion date for immediate Data Flow runs.
• Fixed Object Model fail modal text copies and display issues in AE Flows.
• Resolved error trace formatting and incorrect value display in Data Publishing.