Custom Tasks

🚧

Advanced Option

Note that Custom Tasks are only available if you have Advanced Options enabled. If you do not see this Custom Tasks as an option in your menu, it is because you do not have Advanced Options enabled.

Custom Tasks offer the ability to execute a single notebook within your deployment. This feature allows you to run any logic as needed within that notebook providing added flexibility beyond the specialized task types already available (Acquisition, Transformation, and Publishing Tasks).

Recommendations

While Custom Tasks offer great flexibility, we strongly recommend using the specialized task types (Acquisition, Transformation, and Publishing) whenever possible. These task types come with additional functionalities, such as advanced logging capabilities, that Custom Tasks do not provide.

However, if you need a straightforward way to run a notebook, Custom Tasks are the ideal solution.

How to Use Custom Tasks

NOTE: custom tasks are currently feature flagged in the UI by default. Talk to your delivery team today for how to turn this on for your deployment.

Once custom tasks have been enabled in your environment, you can create a new Custom Task by following these steps:

1: Add: Click on the blue plus sign button to start a new task.

2: Initial Setup: Choose a unique name for your Custom Task and specify the filepath to the notebook located in your Lakehouse then hit Create.

3: Configure: After creation, you will see your new task added to your custom tasks list. Custom Tasks allow you to pass key value pairs to the notebook you want to execute. This is helpful if you want to be able to have a single notebook used for multiple purposes based on some variable parameters. To add these, simply click on "View Configuration" and then "Add new Record" to add your key value pairs.

4: Run: Once added, you can now run the task manually or define a schedule for it to run on automatically. The specified notebook will be executed each time the custom task runs.