Log table changes
Log table Changes:
Old log table structure
job_id STRING COMMENT "The global run id",
run_id STRING COMMENT "The run id",
source_table STRING COMMENT "The source table name",
table_name STRING NOT NULL COMMENT "The table name being migrated",
partition_id INT NOT NULL COMMENT "The data partition id",
batch_id INT NOT NULL COMMENT "The batch id",
batch_size INT NOT NULL COMMENT "The number of records in the batch",
write_time FLOAT NOT NULL COMMENT "The time taken to write the partition",
time TIMESTAMP NOT NULL COMMENT "The timestamp the partition write finished",
error STRING COMMENT "Any error that occurred during the write",
success_inserts INT COMMENT "The number of successful inserts",
success_updates INT COMMENT "The number of successful updates",
no_ops INT COMMENT "The number of no-ops",
failed_inserts INT COMMENT "The number of failed inserts",
failed_updates INT COMMENT "The number of failed updates",
skipped INT COMMENT "The number of failed no-ops",
quarantine_table STRING COMMENT "The table name for quarantined records"
New log table structure:
"log_id": "BIGINT GENERATED ALWAYS AS IDENTITY",
"job_id": "STRING",
"run_id": "STRING",
"source_table": "STRING",
"table_name": "STRING",
"partition_id": "INT",
"partition_size": "INT",
"write_time": "FLOAT",
"success_inserts": "INT",
"success_updates": "INT",
"skipped": "INT",
"failed_inserts": "INT",
"failed_updates": "INT",
"invalid_records": "INT",
"error": "STRING",
"start_time": "TIMESTAMP",
"end_time": "TIMESTAMP",
"quarantine_table": "STRING",
Removed these columns from the log table:
batch_id INT NOT NULL COMMENT "The batch id",
batch_size INT NOT NULL COMMENT "The number of records in the batch",
time TIMESTAMP NOT NULL COMMENT "The timestamp the partition write finished",
no_ops INT COMMENT "The number of no-ops",
Added these new columns to log table:
"partition_size": "INT",
"invalid_records": "INT",
"start_time": "TIMESTAMP",
"end_time": "TIMESTAMP",
Updated 3 months ago