support except table#74
Conversation
w41ter
left a comment
There was a problem hiding this comment.
Please add a regression test for the except tables, and cover the below situations on the except tables:
- normal insert
- add/drop partition
- drop table
- alter/truncate table
- light schema change on the table
| Src base.Spec `json:"src,required"` | ||
| Dest base.Spec `json:"dest,required"` | ||
| SkipError bool `json:"skip_error"` | ||
| ExceptTable string `json:"except_table"` |
There was a problem hiding this comment.
Define it to an array might be better.
| ExceptTable string `json:"except_table"` | |
| ExceptTables []string `json:"except_tables"` |
There was a problem hiding this comment.
Then run strings.TrimSpace(exceptTable) before converting jobContext to Job.
| dest base.Spec | ||
| db storage.DB | ||
| skipError bool | ||
| exceptTable string |
There was a problem hiding this comment.
| exceptTable string | |
| exceptTables []string |
| IDest base.Specer `json:"-"` | ||
| destMeta Metaer `json:"-"` | ||
| SkipError bool `json:"skip_error"` | ||
| ExceptTable string `json:"except_table"` |
There was a problem hiding this comment.
| ExceptTable string `json:"except_table"` | |
| ExceptTables []string `json:"except_tables"` |
| } | ||
| } | ||
|
|
||
| func IsExceptTable(exceptTables string, targetTable string) bool { |
There was a problem hiding this comment.
Define it as a method of Job.
It likes:
func (j *Job) isExceptTable(targetTable string) bool {
|
|
||
| backupTableList = append(backupTableList, table.Name) | ||
| } | ||
| log.Infof("fullsync table is : %s", backupTableList) |
There was a problem hiding this comment.
There may be thousands of tables here, so it is not suitable for printing logs.
| } else if j.SyncType == DBSync { | ||
| // use sourceTableName to judge if table is except | ||
| var sourceTableName string | ||
| sourceTableName, err = j.srcMeta.GetTableNameById(dropPartition.TableId) |
There was a problem hiding this comment.
GetTableNameById is a high-cost operation, it needs to connect doris FE and execute 'SHOW TABLE'.
One possible solution is to query the table ID of the except tables before adding the job to JobManager, see Job::FirstRun for details.
post is
You could specify except_table, that tables will not sync。
BTW, except_table could be "table1, table2,table3", separate by comma