Follow this guide to set up entities, columns, and dependencies for model building using Advanced Options.

This process involves creating new Models, Entities, Entity Columns, and Entity Dependencies in the appropriate tabs in Advanced Options. If you want to learn more about these concepts, visit the pages below:

• Model
• Entity
• Entity Column
• Entity Dependency

🚧

Staging Notebooks

This guide assumes you have already written the appropriate notebooks to stage your data transformations. Follow the Staging Data with Notebooks guide for more details if necessary.

Models

Navigate to the Advanced Options pane on the left side of your screen. On the top of Advanced Options, navigate to the Models tab.

📘

What are models?

Models are the unit of measure for an atomic model building process. When the model building process is complete, the model is the final built list of gold-layer objects to be used for end processes, such as reporting, AI workflows, or uses.

This table displays a list of all models currently in your environment. You can view a model's name, description, and toggle it active or inactive. To create a new model, click on "NEW RECORD"

Fill out the fields for name and description. We recommend using a name that is descriptive of the model's end-purpose. Then click "CREATE" to create the new model.

You can see this new model exists in the Model tab.

👍

Activate your Model!

Make sure to toggle the model ACTIVE in order to start using it. Inactive models will not be built by the Empower system.

Entities

📘

What are Entities?

Entities are the gold-layer tables that are generated during the model building process. You can built any number of entities during a model build. Entities can be built using data from any bronze or silver table in your data estate. They can even be built using other gold-layer entities!

🚧

Know your Staging Notebook Locations!

This section assumes you have your staging notebooks written and locations to those notebooks are known. We will be configuring those locations to the entries in the Entities table, so have those on hand.

The Entity table defines the list of dims and facts for all your models. For each Entity, you can view the model this entity belongs to, its schema, the name of the entity and description, its type (a various assortment of dim and fact types), the source schema and source entity, and the staging notebook that this entity is build within. You can activate and deactivate any entity on demand. Deactivated entities will not be built during a model build process.

Below is a table describing each of the Entity columns and example values.

Create a new entity by clicking on "New Record". This will bring up the new Entity creation modal.

📘

Entities Default to Inactive

After creating a new entity, you will have to toggle it on for it to be built during the model building phase. All entities default to inactive state upon creation.

After filling out fields, clicking "Create" will result in a new entry in this table. Create an entry for each of your staging notebooks/entities in your model.

👍

Activate your Entity!

Make sure to toggle the Entity ACTIVE in order to start using it. Inactive Entities will not be built by the Empower system.

Entity Types

Entities can be several types, depending on their use and what kind of historical tracking is needed.

Understanding Full and Incremental Load Types

When building a model, it's essential to understand the difference between full and incremental load types. This section provides an overview of these two types and their use cases.

Full Loads

A Full Load involves loading all the data from the source to the destination each time the data is processed. In other words, the entire dataset is reloaded, regardless of whether the data has changed or not. This method can be a good choice when:

• The dataset is small, and the time required to process and reload the entire dataset is relatively short.
• The source data does not have a way to track changes or does not support incremental loads. **
• The latency will be roughly equal in both full and incremental, up to the processing time. Incremental will be more "real-time" in most cases.

However, Full Loads can be time-consuming and resource-intensive, especially when dealing with large datasets or frequent updates.

Incremental Loads

Incremental Load, on the other hand, only processes and loads new or modified data since the last load. This method is more efficient, as it reduces the amount of data being transferred and processed. It can be a good choice when:

• The dataset is large, and processing the entire dataset takes a long time.
• The source data has a method to track changes, such as a modified date or timestamp. **
• The data warehouse or destination can tolerate some delay in reflecting the latest changes.

Incremental Loads may require additional logic and setup to identify and process only the new or modified data. They may also require monitoring and handling of potential data conflicts or issues that may arise when merging the new data with the existing data.

Entity Columns

📘

What are entity columns?

Entity columns allow Empower to track and maintain detailed metadata about entities.

Entity columns should be a 1:1 mapping with the fields of all gold-layer dims and facts. All entity columns must be linked to an existing Entity (defined in the previous section above).

Create a new Entity Column record by clicking "New Record". This will bring up the Entity Column creation modal.

Below is a table describing each of the fields in an Entity Column record.

After filling out fields, hitting the create button will result in a new entry in this table.

🚧

Ordinal Column

The ordinal setting is only taken into account during the first model build. Any subsequent build will assume the order within the existing table. In other words, if Ordinal settings are changed after the first time the model is successfully built, these changes will not be respected.

👍

Activate your Entity Column!

Make sure to toggle the Entity Column ACTIVE in order to start using it. Inactive Entity Columns will not be built by the Empower system.

Type ID

Selecting the correct Type ID for your column is as simple as selecting from a drop down list. Please refer to the following table to understand how assignments should be made.

Entity Dependency

📘

What is an Entity Dependency?

Entity Dependencies help Empower define the order in which dims and facts get built and how they can be optimally parallelized. An accurate dependency assignment can be the difference between a fast and efficient model build and a slow or failing process.

Make sure to study your staging logic to determine what entities depend on each other.

Create a new Entity Dependency record by clicking "New Record". This will bring up the Entity Column creation modal.

Recommendations

1. Assign parent-child entity relationships. In such relationships, the child table is dependent on the parent table, and it must wait for the completion of the parent table before proceeding. Make sure to assign parent and child entities correctly, and that the model_id column points to your model.
2. Assign any entities that don't have a prior depency directly to the model itself. To do this: make a new record and leave the parent entity ID blank! Define the child entity to be the that entity, and define the model to be your model.