Setup to run dynamics publishing using spark package

Clone the source query folder and make below changes

  1. Change stage database name in shared_utils.py : mwa_stage to mwa_stage_m

  2. Replace
    client[table_name1] to client.read_table(tab_name1)

  3. Replace

   source = TableSource(f"{source_table_name}")  
   checkpoint_location = os.path.join(migration_root, "checkpoints", table_name)  
   dbutils.fs.rm(checkpoint_location, True)  
   migrator.auto_load(checkpoint_location, data_source=source, table_name = ç, mode="append", 
   key_columns=business_key_list, workers=32)

with

    p_source_table = f"{source_table_name}"
    p_key_columns = business_key_list
    p_target_table = table_name
    p_mode = "append"
    dynamics_client.load(p_source_table, p_key_columns, p_target_table, p_mode)