Skip to content

support except table#74

Open
lsy3993 wants to merge 2 commits intoselectdb:devfrom
lsy3993:lsy_except_table
Open

support except table#74
lsy3993 wants to merge 2 commits intoselectdb:devfrom
lsy3993:lsy_except_table

Conversation

@lsy3993
Copy link
Copy Markdown
Collaborator

@lsy3993 lsy3993 commented May 7, 2024

post is

curl -X POST -H "Content-Type: application/json" -d '{
        "name": "ccr_test1",
        "src": {
        "host": "10.16.10.6",
        "port": "19035",
        "thrift_port": "19025",
        "user": "root",
        "password": "",
        "database": "test",
        "table": ""
        },
	"except_table":"example_range_tbl",
        "dest": {
        "host": "10.16.10.6",
        "port": "19035",
        "thrift_port": "19025",
        "user": "root",
        "password": "",
        "database": "test_bak",
        "table": ""
        }
    }' http://127.0.0.1:9190/create_ccr

You could specify except_table, that tables will not sync。
BTW, except_table could be "table1, table2,table3", separate by comma

Copy link
Copy Markdown
Contributor

@w41ter w41ter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a regression test for the except tables, and cover the below situations on the except tables:

  1. normal insert
  2. add/drop partition
  3. drop table
  4. alter/truncate table
  5. light schema change on the table

Src base.Spec `json:"src,required"`
Dest base.Spec `json:"dest,required"`
SkipError bool `json:"skip_error"`
ExceptTable string `json:"except_table"`
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Define it to an array might be better.

Suggested change
ExceptTable string `json:"except_table"`
ExceptTables []string `json:"except_tables"`

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then run strings.TrimSpace(exceptTable) before converting jobContext to Job.

Comment thread pkg/ccr/job.go
dest base.Spec
db storage.DB
skipError bool
exceptTable string
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
exceptTable string
exceptTables []string

Comment thread pkg/ccr/job.go
IDest base.Specer `json:"-"`
destMeta Metaer `json:"-"`
SkipError bool `json:"skip_error"`
ExceptTable string `json:"except_table"`
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
ExceptTable string `json:"except_table"`
ExceptTables []string `json:"except_tables"`

Comment thread pkg/ccr/job.go
}
}

func IsExceptTable(exceptTables string, targetTable string) bool {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Define it as a method of Job.

It likes:

func (j *Job) isExceptTable(targetTable string) bool {

Comment thread pkg/ccr/job.go

backupTableList = append(backupTableList, table.Name)
}
log.Infof("fullsync table is : %s", backupTableList)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There may be thousands of tables here, so it is not suitable for printing logs.

Comment thread pkg/ccr/job.go
} else if j.SyncType == DBSync {
// use sourceTableName to judge if table is except
var sourceTableName string
sourceTableName, err = j.srcMeta.GetTableNameById(dropPartition.TableId)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GetTableNameById is a high-cost operation, it needs to connect doris FE and execute 'SHOW TABLE'.

One possible solution is to query the table ID of the except tables before adding the job to JobManager, see Job::FirstRun for details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants