Overview

Effective monitoring is key toensuring the reliability and timeliness of your data processing tasks.

Empower provides robust tools for monitoring your data estate, enabling you to track their execution status, performance, and any issues that may arise. From , we detail how users can leverage the flow log page in Empower to monitor their data flows, focusing on aspects such as start times, run IDs, durations, and statuses.

Run Logs

Empower provides logging capabilities to track the various current and historical runs of any data flow.

Navigate to a specific flow's logs by clicking "View Runs" for that flow.

Run ID: Upon initialization a run is assigned a unique Run ID. This identifier is crucial for tracking specific instances of runs programmatically, especially when diagnosing issues or auditing data processing tasks.

Start Times: This indicates when the run execution began. Tracking start times helps in understanding the schedule adherence and identifying any delays in execution.

End Time: This indicates when the run execution completed, either successfully or in failure.

Duration: Duration tracks the length of time a run took to complete. Significant deviations from the norm can indicate problems or inefficiencies in data acquisition, processing, or publishing.

Status: Status (e.g., “Success,” “Failed,” “Running”) provides immediate insight into a run's current or completed state.

Run Details

🚧

Availability

Run details are currently only available for Data Acquisition Flow runs. Analytics Engineering and Publishing Flows will have this detail view available in upcoming Empower releases.

To see additional detail about a specific run, you can click on a specific row in the run log table.

The run detail view provides you with an object-by-object (or entity-by-entity depending on the data flow type) view of everything that happened during that run.

Note that the details shown in this view will vary based on the data flow type. See the data flow type sections below for more detail on what information is shown for each type.

In the case of a failed run, you can drill in to see the exact error that caused the failure by clicking on row of the object that failed.

Data Acquisition Run Details

For Data Acquisition flows, you’ll notice two distinct status bars and columns: one for Extraction and one for Ingestion.

These flows encompass a few steps bundled under the term "acquisition" to simplify the user experience. To provide you with detailed and granular information for easier debugging, we’ve separated and displayed each step individually. Read more about how Data Acquisition works and the steps involved here.

The Data Acquisition run details table provides the following additional

Hovering over the status bar in the Extraction column will provide more details on the number of rows processed for that object and the duration it took to complete.

Similarly for Ingestion, hovering will provide additional details on the duration to complete, rows ingested, and file path.