Empower Data Platform v1.13

👍

Empower UX capability and Data Platform updates

Overview

In the Empower UI, we are moving closer to achieving complete support for Metadata Management. This includes support for metadata extraction/refresh and the capability to view extracted metadata at both the object-level and the field-level. These enhancements provide users with greater visibility into their source metadata, which sets the foundation for effective Data Acquisition (DAQ) flow management.

Meanwhile, we are refactoring connectors to ensure compliance with the new ingestion engine, including the Kronos connector. Our connector library is continuously expanding, offering clients access to a wider range of data sources with every release.

Features

• Metadata Management UI/UX

Changes

• Implemented metadata extraction for stable Empower platform connections, providing users with valuable insights into their data structure.
• Optimize your workflow with the power to initiate manual triggering of metadata extraction pipelines from the Empower UI. Benefit from lightweight monitoring, error detection, and re-triggering capabilities, ensuring seamless operations and high-quality data management.
• Refactored connectors, including Kronos, for compatibility with the forthcoming Spark engine, enabling faster data processing.
• Enhance reliability with an improved connector testing program that ensures robust release of connections. Enjoy a wider range of thoroughly tested and compatible connections for seamless integration.
• Diligently addressed miscellaneous bugs to maintain the stability and reliability of the application. By resolving these issues, we have optimized the overall system performance, resulting in a smoother and more seamless user experience.

User Impacts

• Users can now unlock the full potential of their connected data sources by extracting and visualizing metadata. Gain valuable insights for informed decision-making and optimized data management.

Notes / Coming Soon

• While not yet implemented in this release, we have made substantial progress in completing the ingestion engine upgrades and implementing a Unity Catalog compliant infrastructure. These enhancements will undergo rigorous testing to ensure their reliability and effectiveness. They are scheduled to be rolled out in the next release (version 1.14). Once deployed, clients can anticipate notable improvements, with ingestion speeds expected to increase by 20% or more. As customers are migrated to UC-compliant deployments, they will experience faster and more efficient data ingestion processes.
• Regarding Metadata Management, users can now extract, refresh, and view their metadata, providing them with valuable insights into their data sources. Although the 'Save' functionality is not currently available, it will be implemented in version 1.14. This upcoming feature will allow users to conveniently save their metadata edits, ensuring that any changes or modifications made to the metadata are securely stored and easily accessible for future reference and analysis.