Custom Flows

BETA The Custom Flows feature is currently in BETA mode.

🚧

Advanced Option

Note that Custom Flows are only available if you have Advanced Options enabled. If you do not see this Custom Flows as an option in your menu, it is because you do not have Advanced Options enabled.

Custom Flows offer the ability to execute a single notebook within your deployment. This feature allows you to run any logic as needed within that notebook providing added flexibility beyond the specialized flow types already available (Data Acquisition, Analytics Engineering, and Publishing flows).

Recommendations

While Custom Flows offer great flexibility, we strongly recommend using the specialized flow types (Data Acquisition, Analytics Engineering, and Publishing) whenever possible. These flow types come with additional functionalities, such as advanced logging capabilities, that Custom Flows do not provide.

However, if you need a straightforward way to run a notebook, Custom Flows are the ideal solution.

How to Use Custom Flows

NOTE: custom flows are currently feature flagged in the UI by default. Talk to your delivery team today for how to turn this on for your deployment.

Once custom flows have been enabled in your environment, you can create a new custom flow by following these steps:

1: Add: Click on the blue plus sign button to start a new flow.

2: Initial Setup: Choose a unique name for your custom flow and specify the filepath to the notebook located in your lakehouse then hit Create.

3: Configure: After creation, you will see your new flow added to your custom flows list. Custom flows allow you to pass key value pairs to the notebook you want to execute. This is helpful if you want to be able to have a single notebook used for multiple purposes based on some variable parameters. To add these, simply click on "View Configuration" and then "Add new Record" to add your key value pairs.

4: Run: Once added, you can now run the flow manually or define a schedule for it to run on automatically. The specified notebook will be executed each time the custom flow runs.