Acquire
Pulling data into the lakehouse from external sources
Acquire Tasks (or acquisition tasks) define the inbound movement of data, i.e. data moving from a data source into your centralized, enterprise data Lakehouse.
In Empower, you can create as many acquire tasks as you want, while configuring each of them to acquire data however often you want.
Quick Links
This page only covers an overview of Acquisition Tasks and how to use them
- Logging and Monitoring: defines how to use the log page to monitor tasks ("View Runs").
- Schedules and Triggers: describes how to trigger tasks to run on demand ("Preview Run") and schedule tasks to run on a repeating cadence ("Schedule").
Overview
The Acquire module is accessible from the left navigation menu.
Click Acquire to navigate to the module.
From here, you can view all Acquisition Tasks within your selected environment.
There are a few restrictions when it comes to acquisition tasks.
- An Acquisition Task can only ever be affiliated with one Data Source Connection at a time.
- This means you can't bring data in from multiple data sources under the same task! In case you have this need, create a separate acquisition task for each of your data sources, and schedule them to run at the same time.
- Archiving a Data Source Connection will also force the archiving of all affiliated Acquisition Task.
- You will not be able to edit, schedule, or trigger these tasks until you restore the underlying Data Source Connection.
- Deleting a Data Source Connection will permanently archive all affiliated Acquisition Tasks.
- You will never again be able to edit, schedule, or trigger these tasks.
Tasks
From the default view for the Acquire module, you can see all the Acquisition Tasks within your current environment.
A list of all Acquisition Tasks.
You can search these tasks by name, view configurations or historical run logs, trigger a task to run on demand, view/set/activate scheduling, and create new tasks.
Creation
Creating an Acquisition Task can be done by simply clicking on "+" at the top of the screen on the homepage for the Acquire module.
Create a new Acquisition Task by clicking "+".
Fill out the name of the task and select an existing Data Source Connection from the drop down.
Once you fill out the required fields (Name and Data Source), click "Create" to complete the creation process.
"Create" is now enabled, now that Name and Data Source (required fields) are filled out.
You can now see your newly created task at the top of the page.
Congratulations, you have successfully created a Acquisition Task.
Advanced Features
For a peak at the advanced features you may configure to optimize Acquisition, take a look at our guide on Batch Extraction.
INFO: 1 to Many - Data Source Connections and Acquisition Tasks
Connections and Acquisition Tasks have a 1:many relationship.
This means that a single Acquisition Task can only ever be associated with one Data Source Connection at any moment in time. However, a specific Data Source Connection may be associated with many different Acquisition Tasks, all with different schedules and inclusion configurations.
Editing
You can modify any existing Acquisition Task by clicking "..." near the name of the task. Doing so will bring up a menu with one option being to Edit the task.
Click "..." and then Edit to modify an existing task.
You may edit the Name and the Data Source Connection associated with this task.
WARNING: changing the Data Source Connection field will affect your configuration!
Changing a Acquisition's data source will completely wipe any previous object inclusion configurations. Be mindful of this action when you modify a Acquisition Task!
Make your desired edits, and "Save" your changes when ready.
Deletion
You can delete any existing Acquisition Task by clicking "..." near the name of the task. Doing so will bring up a menu with one option being to Delete the task.
Click "..." and then Delete to delete an existing task.
A confirmation modal will pop up. You must confirm you wish to delete the task in order to complete the deletion process.
Confirm you wish to delete the task and all of its historical logs.
DANGER: Deletion is Permanent
Task deletion is a permanent action. Deleting a task will also remove the entire historical log of that task. You will not be able to reverse a task's deletion, so make sure you actually want to perform this action!
Scheduling and Triggering Tasks
To read about how to schedule and trigger Acquisition Tasks or any other task type, visit Scheduling Data Tasks.
Configuration
Click "View Configuration" on any existing Acquisition Task to visit its configuration page.
The configuration page for the Weekly Sales Acquisition Task.
The configuration page contains a list of all available objects from the task's associated Data Source. You may toggle Include to configure this object to be included as part of data acquisition when this task is triggered.
You may search objects by name, as well as filter objects by Include, Estimated Rows and Schema Name.
When you include/exclude an object on this page, the change is automatically saved to the task's configuration.
WARNING: Global Enablement
The Enable toggle on the object within the Data Source's Metadata Catalog will impact the ability of this task to acquire the object.
If the object is globally disabled it will not be acquired when this task runs.
- You will not be able to include globally disabled objects it in the task.
- Even if the disabled object is already included in the task, it will not be acquired when the task runs.
How Data Acquisition Works
Data Acquisition is an umbrella term that links together three data movement concepts.
Metadata Extraction
Acquisition automatically refreshes the associated data source's metadata to ensure Schema Drift protection. Read up about Metadata Extraction and how Metadata Catalogs work.
Extraction
Extraction defines the process of collecting data from a data source and writing it to the open-source parquet file format in the Empower Landing Zone.
A handful of Empower supported data sources for extraction.
Supported Data Sources in Empower
Empower is configured to support over 200+ data sources for extraction. Some of these are entirely self-serviceable from the UI. Others require more configuration by the Empower delivery team.
For a full list of sources, check out our Source System Support page.. Talk to your delivery manager for more information.
Ingestion
Data ingestion takes the data from the Landing Zone and ingests it into the open-sourced Delta Table format into the Bronze Layer of the Medallion Architecture.
All Empower-ingested data is tracked as part of a Type 2 data contract. This means that all changes made to the data from the original data source, including modifications and deletions, are stored as new data points in the Bronze Layer.
Bronze Layer ingested data is further ingested into the Silver Layer as the most current version of every data point.
Updated 20 days ago
To configure Data Acquisition from Data Sources, learn about how to manipulate the Metadata Catalog.