Overview

Data flows can be triggered to run on-demand or as part of a schedule. Scheduled triggers can be configured to repeat on a user-defined cadence, like daily, weekly, or monthly. Empower provides great flexibility for scheduling cadences, as described below in the Scheduling Triggers section.

📘

Question: What if the flow is already running?

If a flow is triggered to run (either with an on-demand trigger or via scheduled trigger) and the flow is already queued or running, Empower will queue the flow to run again as soon as the current run is complete.

There will not be two instances of the same flow running at the same point in time.

On-demand Triggers

Any flow can be triggered to run on-demand at any moment in time. To do so, navigate to the proper module for the flow (Data Acquisition, Analytics Engineering, or Data Publishing), and find the flow you wish to trigger.

Upon clicking Preview Run for this flow, a modal will appear with a preview of what will happen during this flow run.

Data Acquisition

When previewing a run of a Data Acquisition flow, you will see a list of all the objects set to be included with this flow run and their respective load strategies.

Click "Run" to run this flow. Clicking "Cancel" will abort the triggering process.

Analytics Engineering

When previewing a run of an Analytics Engineering flow, you will see a list of model entities set to be built during this flow run, including their type, source schema and source entity.

Click "Run" to run this flow. Clicking "Cancel" will abort the triggering process.

Data Publishing

When previewing a run of a Data Publishing flow, you will see a list of publish entities set to be published during this flow run, including their source schema and entity.

Click "Run" to run this flow. Clicking "Cancel" will abort the triggering process.

Scheduling Triggers

For data flows that need to run at regular intervals (e.g., hourly, daily, weekly), setting up a recurring schedule automates the process, eliminating the need for manual intervention.

To set up a scheduled trigger, navigate to the proper module for the flow (Data Acquisition, Analytics Engineering, or Data Publishing), and find the flow you wish to trigger.

Viewing a Flow's Schedule

You can view the current schedule of a flow below the "Flow Status" field.

If a flow has no schedule set, the schedule field will read "-".

Disabling a Flow's Schedule

By toggling the Schedule Active field "Off," you can disable a flow's schedule. This will temporarily negate any scheduled triggers for this flow until Schedule Active is toggled back to "On". The flow's schedule will reappear when Schedule Active field is toggled back on.

Setting a Flow's Schedule

To set a flow's schedule (including overwriting existing schedules), click the "Schedule" button.

This will bring up a preview modal identical to the one from "Preview Run."

Clicking "Schedule" will continue the schedule configuration process and bring up a second schedule modal.

Use the Start Date and Start Time fields to define when the schedule is set to begin. Use the Time Zone field to define the timezone for this schedule.

You can use the Repeat On field to define a custom frequency to run this schedule.

You can enable the flow to run by the minute, hour, day, week, or month.

Use the Ends field to define a date where the recurrence will end.

When you are finished configuring the flow's schedule, click "Done." The schedule for this flow will now be set according to the schedule configuration. You will see your changes reflected in the Schedule field below the Flow Status.

Best Practices

Testing: Before setting a recurring schedule, run the data flow on-demand to ensure it executes as expected.

Monitoring: Regularly monitor your scheduled data flows to catch and resolve any issues early. Use the relevant monitoring page for the flow to do so

Documentation: Keep documentation of all scheduled data flows, including their purposes, schedules, and any dependencies. This practice is invaluable for maintenance and troubleshooting.