Introduction

The purpose of the Data Migration, or Configuration Migration, feature is to streamline and automate the process of migrating configurations for data movement from a source environment to a target environment. This will enable users to efficiently transfer configurations between different environments, reducing the risk of manual errors and saving time.

On this page, Empower users will find more information about using the Configuration Migration module: the required roles to perform a configuration migration as well as how to manage migrations (create new, revert, etc.).

Supported Roles

In order to perform a configuration migration, a user must be AT LEAST assigned an Executor role in the source environment and a Writer in the target environment. For example, a user attempting to migrate configuration(s) from a Development (DV) environment to Continuous Deployment (CD), they must be AT A MINIMUM DV Executor and CD Writer.

For more information on Empower roles, please see: Access Control.

Migration Management

Create a New Migration

Users may access the Data Migration module via System Settings. Once in the Data Migration module, users with the required role assignments will be able to select the + button to trigger the creation of a new migration (pictured below).

After selecting the + button, the New Migration form opens to the right of the screen where the user may select their target environment and database assets to migrate (pictured below). There are a couple of important points to take note of here:

Users may only select the target environment for promotion. This is because the source environment is assumed per the current environment that the user is in. To trigger a migration from a different environment, use the navigation breadcrumb in the header bar.
Users that do not have a minimum of Executor access in certain environments will not be able to target those environments for migration. They will not be able to select these in the UI until proper access is provided.

If you decide that you do not need a migration, selecting the X in the upper-right corner of the New Migration form will remove the migration instance

Data Source Migration

Once the target environment is selected, the user must determine the asset type(s) included in the migration: Connections, Metadata, Models, or Tasks. At the time of this writing, these asset types are handled differently on the backend and therefore must be handled separately.

When migrating Connection assets, a call is made to the target environment to determine which data source(s) the source and target environments have in common. That way, users may only select the sources which do not already exist in the target environment - preventing the migration of duplicate sources.

🚧
Connection migration nuances:

Preview file generation is not possible; users can only migrate after selecting the desired source(s).

Tracking migration history is also not [yet] possible for Data Source assets - so there will not be a way to go back to see when/who migrated data source assets.

Dropbox Data: because 'Dropbox'/Filedrop source types are treated differently on the Empower platform than other typical data sources, the promotion of such source types is out of scope for the initial release(s) of this feature. In the meantime, Dropbox source types will be promoted manually.

After migrating a Fivetran connection, you must go into the target environment and authorize the connection before you can migrate any associated Tasks or Workflows.

📘
Archived Data Sources
You may migrate archived Data Sources without having to restore them!

Migrating Other Assets

In addition to Connections, you can also migrate the following:

Metadata: migrates the Metadata associated with a Connection.
Models: migrates the selected Model(s) including it's Entities, Entity Columns, and any Entity Dependencies.
Tasks: allows you to migrate Acquire, Tranform and Extend Task types between environments. A few important notes on migrating Tasks:
- Only one Task type can be migrated at a time.
- You must first migrate any Connection associated with a Task if not already done.
- Migrating a Task will also migrate the associated Metadata and Model, if applicable, with them.
- Advanced Options must be enabled in your environment in order to migrate an Acquire Task that is associated with a Fivetran Connection.
Workflows: Advanced Options must be enabled in your environment in order to migrate a Workflow. Migrating a Workflow will also migrate the Task(s) within them, however, associated Connections must still be done separately first.

Unlike migrating Connection assets, migrating other assets offers a lot more features and flexibility to Empower users such as:

Allowing users to download and review the change log before committing to a migration
As such, users can opt not to continue with a migration (Discard Migration) if they catch something while reviewing the changelog
Historic record of the migration exists indefinitely, allowing users to look back at previous migrations and changelogs
Reverting previous migrations

The next 4 sections/subsections will go more in depth on the points above.

Generate Preview File

In migrating Tasks, selecting at least one asset from the list will enable the Preview + Migrate and Migrate buttons for selection (pictured below).

Selecting Preview + Migrate will generate the preview file (pictured below). This may take a few seconds to a couple minutes, depending on how many assets are in the migration. Users may navigate to other areas of the application while this is taking place.

Once the changelog rendering is complete, the status of the migration instance updates to Preview Ready. Navigating to the migration details screen (pictured below), users can perform the following actions:

Download Preview Changes Report: Downloads a .zip file which users can open (outside of the application) to review the changes.
Discard Migration: Discards the migration - instance is removed completely from the UI as no changes are committed.
Complete Migration: Triggers the actual migration. This is discussed more in depth in the next section.

Complete Migration

After a user has reviewed the preview changelog and is satisfied with the proposed changes to the target environment, they can select Complete Migration to trigger the actual migration. Upon doing so, the status of the migration updates to Completing Migration (pictured below), which just as in preview generation, the user may navigate to another area of the application while this is taking place.

NOTE: If users skip the preview file generation step, selecting the Migrate button will yield the same expected behavior.

Once the migration successfully completes, the migration status updates to Complete and record of the migration exists for historical reference in the Data Migration module (pictured below).

Revert Migration

After successful 'Other Assets' migrations, users may revert those migrations if they notice any issues with the target environment and suspect that it was a result of the migration. Since historical record is kept for such migrations, users may find the record they want to revert from the list view, then select Revert in the upper-right corner of the migration details screen (pictured below).

Upon selecting Revert, a confirmation modal with ask the user confirm the reversion (pictured below).

If the user is sure they want to revert, selecting Revert in the confirmation modal triggers the revert process (pictured below) - which will DELETE the StateDB records INSERTED during the migration. It is important to note that any successful migration can be reverted at any time, so users need be mindful of the potential for data integrity issues if reverting a migration that already has subsequent successful migrations.

Once the revert is successful, the status of the migration instance updates to Reverted (pictured below). Notice a couple of things updated on the migration details:

The title of the migration instance updates to from 'Migration' to 'Reversion'
If preview changelog was on the original migration, that will no longer be visible or accessible