Changelog
Back to All

Empower Data Platform v1.14

over 1 year ago by [email protected]

👍

Empower UX capability and Data Platform updates

Overview

Get ready for the exciting Empower v1.14 release, packed with updates and improvements:

  • Data Catalog: Redesigned data pipeline & Unity Catalog integration for end-to-end Lakehouse visibility, security, and governance
  • Metadata Management: Edit source metadata with validation logic; full system expansion coming soon
  • Model Builder 2.0: 50% faster data modeling performance - no effort required
  • Internal Data Sharing: Grant permissions for seamless data sharing within your organization
  • Delta Sharing with Partners: Instant, secure data sharing between business partners
  • Enterprise Chat: Advanced AI chat tools like OpenAI GPT4, privately & securely ($995/month)
  • Generative AI for Developers: Full LLM lifecycle support with Databricks' Lakehouse AI capabilities
  • Improved Ingestion Engine: 20% faster pipeline ingestion & further enhancements in August

Features

  • Metadata Management (UI)
  • Advanced Options - State DB Tables Edit Grid (UI)
  • Unity Catalog - Online
    • Data sharing within organization
    • Data sharing between partners/other tenants
    • Large Language Model lifecycle support
  • Model Builder 2.0
  • Improved Ingestion Engine
  • Enterprise Chat ($995/month)

Changes

  • Metadata Management:
    • Row validation logic has been implemented for the Metadata Catalog UI
    • Auto-save functionality on the Metadata Catalog
  • Advanced Options - State DB Tables Edit Grid:
    • Created new API endpoints to fretch State DB tables:
      • DatabaseToStepCommand
      • Entity
      • EntityColumn
      • EntityDependency
      • Model
    • Implemented UI access points: included Advanced Options toggle and table edit grid

What's New

Empower v1.14 is synonymous with Unity Catalog functionality, in addition to a host of other improvements! Our delivery teams will begin updating customers to take advantage of this release, with unified security and governance, new UI functionality, and data acquisition upgrades.

No need to copy data within your organization for other Lakehouse’s, Empower environments, or products. Now you simply grant permission and go with Unity Catalog. No cloning, no pipelines, no hassle. Empower customers can also now share or even buy and sell data between business partners: updates are instantly available from the source with no latency, meaning that you can share (or use shared data) the moment it lands in the source table.

Every customer with v1.14 will now take advantage of the ingestion engine upgrades that come with the new Unity Catalog compliant infrastructure. We anticipate significant speed improvements during every ingestion run (over 20%!) and every model build (over 50%!) as a result of these changes.

From the UI-side, all Empower customers will benefit from new features added to this release. As part of the Metadata Management feature, users now have full edit and auto-save functionality on their extracted source metadata. As Empower users could be updating hundreds, if not thousands of rows of metadata, auto-save will aid in keeping track of their changes in real-time as they make them. Should they lose their session or navigate away, their changes will be saved indefinitely until they decide to update them again.

In addition to Metadata Management, UI users will also see access to the Advanced Options feature--which will provide UI visibility of the underlying StateDB tables that are powering their data platform. With this added visibility, more advanced users will be able to view the data pipelines configured to run in their implementations as well as better diagnose any errors in their data platform(s).

Empower is proud to release a new offering in 1.14: Enterprise Chat. This new offering can be purchased separately from Empower to enable organizations to use powerful chatbots like GPT4 in a secure, private way.

Using Unity Catalog features, Empower builds upon this by enabling organizations full Large Language Model lifecycle support. Train off-the-shelf open source models on your own datasets; you already own the data, now you can own your own AI model.

User Impacts

  • Metadata Management:
    • Users can not only make edits to their extracted source metadata, but now there is row validation logic in place to better guid users through the process and surface errors when they are making invalid updates
    • Auto-save functionality has also been included in this release, which puts less cognitive load on users to track source metadata updates - that way if they unexpectedly get logged out or lose their login session, they can pick up where they left off when they return to the Metadata Catalog
  • Advanced Options:
    • Users now have read-only access to the tables in StateDB mentioned above. These tables are critical to the day-to-day operations of the Empower system. Giving users visibility to these tables gives them a better understanding of their platform, the pipelines operating under the hood, as well as surfacing any pipeline run issues
  • Unity Catalog:
    • Unity Catalog brings a unified security and governance feature set to Empower. This means that users can now build a central governance model for all tables, files, and more within the platform! Users can define row filters and column masks for different users and groups, giving finer-grain control needed to keep enterprise data secure without losing utility!
  • Model Builder 2.0:
    • UC requries the user of the new Model Builder 2.20 service within Empower, bringing about better data modeling with large build speedups, over 50% faster!
  • Ingestion Engine Optimizations:
    • UC also comes with ingestion engine improvements. Users an expect an average of ~20% speedup during the ingestion phase once 1.14 is fully deployed!