Setup to run dynamics publishing using spark package
Clone the source query folder and make below changes
-
Change stage database name in shared_utils.py : mwa_stage to mwa_stage_m
-
Replace
client[table_name1] to client.read_table(tab_name1) -
Replace
source = TableSource(f"{source_table_name}")
checkpoint_location = os.path.join(migration_root, "checkpoints", table_name)
dbutils.fs.rm(checkpoint_location, True)
migrator.auto_load(checkpoint_location, data_source=source, table_name = ç, mode="append",
key_columns=business_key_list, workers=32)
with
p_source_table = f"{source_table_name}"
p_key_columns = business_key_list
p_target_table = table_name
p_mode = "append"
dynamics_client.load(p_source_table, p_key_columns, p_target_table, p_mode)
Updated 4 months ago