Setup to run dynamics publishing using spark package
Clone the source query folder and make below changes
- 
Change stage database name in shared_utils.py : mwa_stage to mwa_stage_m
 - 
Replace
client[table_name1] to client.read_table(tab_name1) - 
Replace
 
   source = TableSource(f"{source_table_name}")  
   checkpoint_location = os.path.join(migration_root, "checkpoints", table_name)  
   dbutils.fs.rm(checkpoint_location, True)  
   migrator.auto_load(checkpoint_location, data_source=source, table_name = ç, mode="append", 
   key_columns=business_key_list, workers=32)
with
    p_source_table = f"{source_table_name}"
    p_key_columns = business_key_list
    p_target_table = table_name
    p_mode = "append"
    dynamics_client.load(p_source_table, p_key_columns, p_target_table, p_mode)Updated 7 days ago
