Model

The Model module is accessible by clicking on Model from the navigation pane.

From the Models module, you can view and manage all models within your selected environment. A model holds all the information and configuration necessary to successfully transform data into analytical models.

📘

Question: How are models used?

Models are the unit of measure for an atomic model building process. When the model building process is complete, the model is the final built list of gold-layer objects to be used for end processes, such as reporting, AI workflows, or uses.

Every Transformation Task must be associated with one model. The Entity, Entity Columns, and Entity dependency columns for that model will then appear within that task's Configuration.

🚧

WARNING: A model change will be reflected for all associated tasks!

For example: Task A and Task B are both associated with Model P.

If a user makes changes to an Entity within Task A's configuration, those changes will also take effect for Task B's configuration, as they share the same Model P.

This table displays a list of all models currently in your environment. You can view a model's name, description, and toggle it active or inactive.

Creation

To create a new model, click on "New Model".

This will bring up the model creation blade. Fill out the name (required) and the description (optional) before clicking create.

You can see this new model exists in the Model table.

Overview

Click into a model to view its overview and configuration screens. The default page for any model is the Overview screen. Here you can view and edit name/description information, as well as view any Transform Tasks affiliated with this model.

Entities

📘

What are Entities?

Entities are the gold-layer tables that are generated during the model building process. You can built any number of entities during a model build. Entities can be built using data from any bronze or silver table in your data estate. They can even be built using other gold-layer entities!

🚧

Know your Staging Notebook Locations!

This section assumes you have your staging notebooks written and locations to those notebooks are known. We will be configuring those locations to the entries in the Entities table, so have those on hand.

The Entity table defines the list of dims and facts for the Transformation Task's model. For each Entity, you can view its schema, the name of the entity and description, its type (a various assortment of dim and fact types), the source schema and source entity, and the staging notebook that this entity is built within. You can activate and deactivate any entity on demand. Deactivated entities will not be built during a model build process.

Below is a table describing each of the Entity columns and example values.

Creation

Create a new entity by clicking on "New Record". This will bring up the new Entity creation modal.

📘

Entities Default to Inactive

After creating a new entity, you will have to toggle it on for it to be built during the model building phase. All entities default to inactive state upon creation.

After filling out fields, clicking "Create" will result in a new entry in this table. Create an entry for each of your staging notebooks/entities in your model.

👍

Activate your Entity!

Make sure to toggle the Entity ACTIVE in order to start using it. Inactive Entities will not be built by the Empower system.

Modification

You can modify many of the fields for an entity by clicking into the cell you wish to edit. Press the ENTER key on your keyboard or click away from the cell to save your changes.

Deletion

You can multiselect entities from this table, then click "Delete" in the lower left corner of the screen to delete these entities.

You must confirm that you recognize the results of this action before proceeding.

Entity Types

Entities can be several types, depending on their use and what kind of historical tracking is needed.

When building a model, it's essential to understand the difference between full and incremental load types. This section provides an overview of these two types and their use cases.

Full Loads

A Full Load involves loading all the data from the source to the destination each time the data is processed. In other words, the entire dataset is reloaded, regardless of whether the data has changed or not. This method can be a good choice when:

• The dataset is small, and the time required to process and reload the entire dataset is relatively short.
• The source data does not have a way to track changes or does not support incremental loads. **
• The latency will be roughly equal in both full and incremental, up to the processing time. Incremental will be more "real-time" in most cases.

However, Full Loads can be time-consuming and resource-intensive, especially when dealing with large datasets or frequent updates.

Incremental Loads

Incremental Load, on the other hand, only processes and loads new or modified data since the last load. This method is more efficient, as it reduces the amount of data being transferred and processed. It can be a good choice when:

• The dataset is large, and processing the entire dataset takes a long time.
• The source data has a method to track changes, such as a modified date or timestamp. **
• The data warehouse or destination can tolerate some delay in reflecting the latest changes.

Incremental Loads may require additional logic and setup to identify and process only the new or modified data. They may also require monitoring and handling of potential data conflicts or issues that may arise when merging the new data with the existing data.

Entity Columns

📘

What are entity columns?

Entity columns allow Empower to track and maintain detailed metadata about entities.

📘

Coming Soon: Entity Filters!

We will be adding the ability to filter the entity column table in a future version of Empower coming to you soon! Stay tuned!

Entity columns should be a 1:1 mapping with the fields of all gold-layer dims and facts. All entity columns must be linked to an existing Entity.

Below is a table describing each of the fields in an Entity Column record.

📘

Missing Values and Ghost Records

Empower uses ghost records to gracefully handle missing values in entity columns. In a Kimball data warehouse model, a ghost record (also known as a "dummy record" or "default record") is used to handle scenarios where there are missing or unknown values in a dimension table.

Ghost records are particularly useful in scenarios where:

• Data comes from multiple sources, and some sources might have incomplete or inconsistent data.
• The data quality is not guaranteed, and there might be missing or unknown dimension values.
• The business requires every transaction to be accounted for, even if some attributes are unknown.

By implementing ghost records, you can create a more resilient and user-friendly data warehouse that can handle real-world data imperfections gracefully.

Creation

Create a new Entity Column record by clicking "New Record". This will bring up the Entity Column creation modal.

After filling out fields, hitting the create button will result in the creation of a new Entity Column.

🚧

Ordinal Column

The ordinal setting is only taken into account during the first model build. Any subsequent build will assume the order within the existing table. In other words, if Ordinal settings are changed after the first time the model is successfully built, these changes will not be respected.

👍

Activate your Entity Column!

Make sure to toggle the Entity Column ACTIVE in order to start using it. Inactive Entity Columns will not be built by the Empower system.

Type ID

Selecting the correct Type ID for your column is as simple as selecting from a drop down list. Please refer to the following table to understand how assignments should be made.

Entity Dependencies

📘

What is an Entity Dependency?

Entity Dependencies help Empower define the order in which dims and facts get built and how they can be optimally parallelized. An accurate dependency assignment can be the difference between a fast and efficient model build and a slow or failing process.

Make sure to study your staging logic to determine what entities depend on each other.

Use entities to build the dependent relationships for entities in your model.

Creation

Create a new graph node by clicking "New Record". This will bring up the node creation modal, where you can select from a list of the models entities to add as a node. Note: you cannot add the same entity more than once to a graph!

Assign entity relationships for your model entities. In such relationships, children nodes are dependent on their parent nodes, and it must wait for the completion of all of their parents before being built themselves. Make sure to assign parent and child nodes correctly.

📘

An entity must be present on the graph to be built!

An entity must be present on the graph to be built, even if it has no dependency relationships. If an entity is not present on your dependency graph, it will not be built during model buildout.

You can create nodes for entities that do not have any dependency relationships. To do this: simply make a new node on the graph for that entity and don't assign any relationships.

Table View

Toggle the table view for Entity Dependency using the Entity Dependency Table toggle.