[Enhancement] add support for restore to ccr#303

Open

Vallishp wants to merge 358 commits into

selectdb:devfrom

Vallishp:restore_binlog

This pull request is big! We're only showing the most recent 250 commits

Commits on Jan 11, 2025

Fix case db_sync by send TEST_* query to target (#52)

w41ter
authored and
Vallishp
committed
Fix case db_sync by send TEST_* query to target (#55)

w41ter
authored and
Vallishp
committed
fix sync db error when table name is keyword (#53)

lsy3993
authored and
Vallishp
committed
add option to start pprof server (#51)

lsy3993
authored and
Vallishp
committed
skip view when set binlog.enable (#50)

lsy3993
authored and
Vallishp
committed
Make the constants configurable (#49)

authored and
Vallishp
committed
fix start helper parameter (#57)

lsy3993
authored and
Vallishp
committed
Add golang workflows

w41ter
authored and
Vallishp
committed
Drop the table if restore is cancelled by signature not matched (#58)

w41ter
authored and
Vallishp
committed
Update changelog after 2.0.3.8

w41ter
authored and
Vallishp
committed
Fix bloomfilter index case by wait add index finish (#56)

w41ter
authored and
Vallishp
committed
Fix materialized view

w41ter
authored and
Vallishp
committed
fix table key word (#60)

lsy3993
authored and
Vallishp
committed
fix syntax of auto partition in db sync test (#61)

lsy3993
authored and
Vallishp
committed
Add tarball target

w41ter
authored and
Vallishp
committed
Release 2.0.3.9

w41ter
authored and
Vallishp
committed
Add LICENSE (#63)

w41ter
authored and
Vallishp
committed
Save key time point for ccr job progress (#64)

w41ter
authored and
Vallishp
committed
Fix keyword name in ADD PARTITION (#65)

w41ter
authored and
Vallishp
committed
Fix drop table with keyword (#67)

w41ter
authored and
Vallishp
committed
Fix drop table keyword name if table schema conflicts (#68)

w41ter
authored and
Vallishp
committed
Fix test keyword name test (#69)

w41ter
authored and
Vallishp
committed
create a handler to view job details (#66)

authored and
Vallishp
committed
Fix test inverted index (#70)

w41ter
authored and
Vallishp
committed
Fix keyword name in job.go (#71)

w41ter
authored and
Vallishp
committed
create a handler can get job progress (#72)

lsy3993
authored and
Vallishp
committed
Avoid restart full sync when network interrupt or process crash (#75)

w41ter
authored and
Vallishp
committed
Update CHANGELOG for 2.1.3

w41ter
authored and
Vallishp
committed
Add platform as tarball suffix

w41ter
authored and
Vallishp
committed
Add drop partition usercase

w41ter
authored and
Vallishp
committed
add more desc about operations (#78)

lsy3993
authored and
Vallishp
committed
Add postgresql as meta db (#77)

lsy3993
authored and
Vallishp
committed
Fix job progress key time point (#80)

w41ter
authored and
Vallishp
committed
Fix request redirection (#81)

authored and
Vallishp
committed
Fix could not found partition id after drop partition (#82)

w41ter
authored and
Vallishp
committed
add ccr variant case (#83)

XueYuhai
authored and
Vallishp
committed
add mtmv case (#84)

lsy3993
authored and
Vallishp
committed
Move sql related operations into Specer (#85)

w41ter
authored and
Vallishp
committed
Fix binlog lost (#86)

w41ter
authored and
Vallishp
committed
Fix add partition sql and add adding partition tests (#88)

w41ter
authored and
Vallishp
committed
Add adding partition test (#89)

w41ter
authored and
Vallishp
committed
remove mtmv case (#90)

lsy3993
authored and
Vallishp
committed
add acid case (#87)

lsy3993
authored and
Vallishp
committed
Save prev commit seq if progress is done (#91)

w41ter
authored and
Vallishp
committed
Correct add list partition sql (#92)

w41ter
authored and
Vallishp
committed
fix acid case (#93)

lsy3993
authored and
Vallishp
committed
Change default connect & rpc timeout to 10s,30s (#94)

w41ter
authored and
Vallishp
committed
add connect and rpc timeout in start (#95)

lsy3993
authored and
Vallishp
committed
remove acid table test (#96)

lsy3993
authored and
Vallishp
committed
Filter temp partition upserts and add insert overwrite case (#97)

w41ter
authored and
Vallishp
committed
Fix add partition case (#98)

w41ter
authored and
Vallishp
committed
Fix add partition sql (#99)

w41ter
authored and
Vallishp
committed
Fix insert overwrite cases (#101)

w41ter
authored and
Vallishp
committed
fix create/drop view bug (#100)

authored and
Vallishp
committed
Remove useless CreateTable (#102)

w41ter
authored and
Vallishp
committed
Log snapshot meta size (#103)

w41ter
authored and
Vallishp
committed
Check backup/restore state for test insert overwrite case (#105)

w41ter
authored and
Vallishp
committed
Update CHANGELOG.md (#110)

authored and
Vallishp
committed
add view and mv case (#104)

lsy3993
authored and
Vallishp
committed
filter dropped table when restore (#115)

authored and
Vallishp
committed
fix view case (#116)

lsy3993
authored and
Vallishp
committed
Fix add partition case by check the partition item (#118)

w41ter
authored and
Vallishp
committed
Filter dropped partitions instead of fullsync (#117)

w41ter
authored and
Vallishp
committed
Make AddPartition record compatibile with alternative json name (#121)

w41ter
authored and
Vallishp
committed
Refactor suites layout (#122)

w41ter
authored and
Vallishp
committed
get jobs from GetAllData (#120)

lsy3993
authored and
Vallishp
committed
Filter dropped tables (#123)

w41ter
authored and
Vallishp
committed
Update CHANGELOG.md

w41ter
authored and
Vallishp
committed
Check job state before update progress (#124)

w41ter
authored and
Vallishp
committed
Support partial snapshot to reduce fullsync overhead (#125)

w41ter
authored and
Vallishp
committed
fix no lag when get_lag successed (#126)

lsy3993
authored and
Vallishp
committed
Fix replace partial partition test case assert content (#127)

w41ter
authored and
Vallishp
committed
Fix cases (#129)

w41ter
authored and
Vallishp
committed
Restore snapshot with clean_restore during fullsync (#128)

w41ter
authored and
Vallishp
committed
use full argument (#130)

lsy3993
authored and
Vallishp
committed
Fix incorrect table mapping for dropped tables (#132)

w41ter
authored and
Vallishp
committed
Improve binlog logs (#133)

w41ter
authored and
Vallishp
committed
Fix downstream db in suites (#134)

w41ter
authored and
Vallishp
committed
Improve ci action (#135)

w41ter
authored and
Vallishp
committed
SHOW RESTORE RESULT

w41ter
authored and
Vallishp
committed
Allow create ccr job even if the target table alrady exists (#136)

w41ter
authored and
Vallishp
committed
Move table to recycle bin if the signature is not matched (#137)

w41ter
authored and
Vallishp
committed
Update CHANGELOG.md

w41ter
authored and
Vallishp
committed
Add schema change detail cases (#143)

w41ter
authored and
Vallishp
committed
Add partial sync table name check (#144)

w41ter
authored and
Vallishp
committed
Make clearing meta cache clearer (#145)

w41ter
authored and
Vallishp
committed
Support partial sync with table alias (#148)

w41ter
authored and
Vallishp
committed
Fix job progress test (#150)

w41ter
authored and
Vallishp
committed
Add allow_table_exists test (#146)

w41ter
authored and
Vallishp
committed
Fix creating view and getting views related to specific table (#149)

smallx
authored and
Vallishp
committed
Support partial sync during schema change (#151)

w41ter
authored and
Vallishp
committed
Support rename operator (#147)

lsy3993
authored and
Vallishp
committed
Drop view if the schema is not matched (#152)

w41ter
authored and
Vallishp
committed
Disable clean tables & partitions by default (#153)

w41ter
authored and
Vallishp
committed
Fix test_add_partition case (#154)

w41ter
authored and
Vallishp
committed
Skip variant test in doris 2.0 (#155)

w41ter
authored and
Vallishp
committed
Skip inverted index with unique mor value (#156)

w41ter
authored and
Vallishp
committed
Skip some cases doris 2.0 not supported yet (#157)

w41ter
authored and
Vallishp
committed
Skip db sync rename table in 2.0/2.1 (#158)

w41ter
authored and
Vallishp
committed
Make schema change cases compatible with 2.0 (#159)

w41ter
authored and
Vallishp
committed
Skip AUTO PARTITION in doris 2.0 (#160)

w41ter
authored and
Vallishp
committed
Allow table exists not support 2.1

w41ter
authored and
Vallishp
committed
Delete sync job before test keyword name

w41ter
authored and
Vallishp
committed
2.0 not support INSERT OVERWRITE yet

w41ter
authored and
Vallishp
committed
2.0 default value has \"\"

w41ter
authored and
Vallishp
committed
Fix view already exists

w41ter
authored and
Vallishp
committed
Delete job first for db sync suites

w41ter
authored and
Vallishp
committed
Fix test keyword name

w41ter
authored and
Vallishp
committed
Fix view and mv test

w41ter
authored and
Vallishp
committed
Disable clean tables & partitions by default (#161)

w41ter
authored and
Vallishp
committed
Fix view and mv test

w41ter
authored and
Vallishp
committed
Handle table not found error during begin txn (#162)

w41ter
authored and
Vallishp
committed
Clear the staled table mapping if dest table is not found (#163)

w41ter
authored and
Vallishp
committed
Rollback job progress when table is not found (#164)

w41ter
authored and
Vallishp
committed
Sync thrift with doris master (#165)

w41ter
authored and
Vallishp
committed
Add feature atomic restore (#166)

w41ter
authored and
Vallishp
committed
Add force_fullsync http API (#167)

w41ter
authored and
Vallishp
committed
Disable test sync view twice in 2.0/2.1

w41ter
authored and
Vallishp
committed
Support drop view (#169)

w41ter
authored and
Vallishp
committed
Drop view if it exists before creating table (#170)

w41ter
authored and
Vallishp
committed
Fix create table failed message (#171)

w41ter
authored and
Vallishp
committed
Update CHANGELOG.md

w41ter
authored and
Vallishp
committed
Simplify regression suites (#172)

w41ter
authored and
Vallishp
committed
Fix partial snapshot table alias in db sync mode (#173)

w41ter
authored and
Vallishp
committed
Add db replace partition test (#174)

w41ter
authored and
Vallishp
committed
Add http api /features to list the feature flags (#175)

w41ter
authored and
Vallishp
committed
Fix error when hyphen in table name (#168)

lsy3993
authored and
Vallishp
committed
Fix table alias when handle lightning schema change (#176)

w41ter
authored and
Vallishp
committed
Support rename column (#139)

authored and
Vallishp
committed
Add table sync alias case (#177)

w41ter
authored and
Vallishp
committed
Refresh dest table id after partial sync (#178)

w41ter
authored and
Vallishp
committed
Support table alias instead of drop during fullsync (#179)

w41ter
authored and
Vallishp
committed
Compatible REPLACE_IF_NOT_NULL default value (#180)

w41ter
authored and
Vallishp
committed
Add mem/goroutine/jobs monitor (#181)

w41ter
authored and
Vallishp
committed
Log timestamp add milliseconds (#182)

w41ter
authored and
Vallishp
committed
Fix nil pointer of job.progress (#183)

w41ter
authored and
Vallishp
committed
Filter dropped indexes to avoid META NOT FOUND (#185)

w41ter
authored and
Vallishp
committed
Fix partial sync with incremental data (#186)

w41ter
authored and
Vallishp
committed
Filter shadow indexes upsert (#187)

w41ter
authored and
Vallishp
committed
Update CHANGELOG.md

w41ter
authored and
Vallishp
committed
Fix test_column_ops and row_storage case

w41ter
authored and
Vallishp
committed
Fix test_column_ops test

w41ter
authored and
Vallishp
committed
Fix create dropped view (#188)

w41ter
authored and
Vallishp
committed
Remove useless log

w41ter
authored and
Vallishp
committed
Update CHANGELOG.md

w41ter
authored and
Vallishp
committed
Enable features

w41ter
authored and
Vallishp
committed
Fix test_filter_dropped_indexes.groovy

w41ter
authored and
Vallishp
committed
Add db sync truncate table case (#190)

w41ter
authored and
Vallishp
committed
Filter the committed binlogs (#191)

w41ter
authored and
Vallishp
committed
Fix test_truncate_table and test_db_sync_signature_not_matched (#192)

w41ter
authored and
Vallishp
committed
fix test_db_sync_signature_not_matched (#193)

w41ter
authored and
Vallishp
committed
Reduce suites name size (#194)

w41ter
authored and
Vallishp
committed
Limit Backend ingesting concurrency (#195)

w41ter
authored and
Vallishp
committed
Add mysql max allowed packet flag (#196)

w41ter
authored and
Vallishp
committed
Support modify comment (#140)

smallx
authored and
Vallishp
committed
Cancel running backup/restore jobs before fullsync (#197)

w41ter
authored and
Vallishp
committed
Avoid lock job when get job status (#198)

w41ter
authored and
Vallishp
committed
Add cancel_conflict_backup_restore_job flag (#199)

w41ter
authored and
Vallishp
committed
Cancel restore job only when feature flag open (#200)

w41ter
authored and
Vallishp
committed
Avoid blocking job in full/partial sync (#201)

w41ter
authored and
Vallishp
committed
skip external table when set binlog enable

lsy3993
authored and
Vallishp
committed
spell error

lsy3993
authored and
Vallishp
committed
Skip unsupported tables during fullsync

w41ter
authored and
Vallishp
committed
Avoid persisting snapshot job info and meta in job progress (#204)

w41ter
authored and
Vallishp
committed
Build table mapping from snapshot job info (#205)

w41ter
authored and
Vallishp
committed
Fix fullsync with alias (#207)

w41ter
authored and
Vallishp
committed
Handle the binlogs wrapped in barrier log (#208)

w41ter
authored and
Vallishp
committed
Support rename table in 2.0/2.1 (#209)

w41ter
authored and
Vallishp
committed
Support rename column in 2.0/2.1 (#210)

w41ter
authored and
Vallishp
committed
Fix infinite loop when backup/restore job failed (#206)

w41ter
authored and
Vallishp
committed
Avoid logging empty db error, wait instead (#211)

w41ter
authored and
Vallishp
committed
Fix add partition with keyword name (#212)

w41ter
authored and
Vallishp
committed
Skip modify partitions binlog (#213)

w41ter
authored and
Vallishp
committed
Fix column ops suite

w41ter
authored and
Vallishp
committed
Skip tmp partition dropping (#214)

w41ter
authored and
Vallishp
committed
Remove readOnly option for mysql (#215)

w41ter
authored and
Vallishp
committed
Reuse the backup/restore task issued by the job itself (#218)

w41ter
authored and
Vallishp
committed
Reorg db/table sync suites (#219)

w41ter
authored and
Vallishp
committed
Fix test_ts_table_modify_comment case (#222)

w41ter
authored and
Vallishp
committed
Support compressed snapshot to avoid thrift max message size limitation (#223)

w41ter
authored and
Vallishp
committed
Enable feature_reuse_running_backup_restore_job by default (#224)

w41ter
authored and
Vallishp
committed
Fix reusing wrong backup/restore job (#226)

w41ter
authored and
Vallishp
committed
Fix suites (#227)

w41ter
authored and
Vallishp
committed
Add test for ccr sync rename partition name (#221)

wyxxxcat
authored and
Vallishp
committed
[test] Add test for db/tbl sync rename table column(#225)

wyxxxcat
authored and
Vallishp
committed
Support snapshot expiration (#229)

w41ter
authored and
Vallishp
committed
Fix drop partition with keyword name (#231)

w41ter
authored and
Vallishp
committed
Correct use of target_sql (#230)

wyxxxcat
authored and
Vallishp
committed
Fix helper.is_version_supported (#233)

w41ter
authored and
Vallishp
committed
Avoid using index name to build base index mapping (#235)

w41ter
authored and
Vallishp
committed
Add flag to skip rollup binlogs (#236)

w41ter
authored and
Vallishp
committed
Fix parallel creating table and fullsync (#237)

w41ter
authored and
Vallishp
committed
Revert "Fix drop partition with keyword name (#231)" (#238)

w41ter
authored and
Vallishp
committed
Fix wrong table name dependencies (#239)

w41ter
authored and
Vallishp
committed
Fix unknown table/partitions in partial snapshot (#240)

w41ter
authored and
Vallishp
committed
Skip overwritten tables in partial snapshot (#241)

w41ter
authored and
Vallishp
committed
Fix wrong test for rename partition (#243)

wyxxxcat
authored and
Vallishp
committed
add tests for schema change

wyxxxcat
authored and
Vallishp
committed
Support replace table binlog (#245)

w41ter
authored and
Vallishp
committed
Check connection error (#247)

w41ter
authored and
Vallishp
committed
Filter the drop missing table binlog (#248)

w41ter
authored and
Vallishp
committed
Support drop view (#138)

smallx
authored and
Vallishp
committed
Fix partially committed replace table binlog (#249)

w41ter
authored and
Vallishp
committed
Support Modify ViewDef binlog (#184)

authored and
Vallishp
committed
Support inverted index binlog (#252)

w41ter
authored and
Vallishp
committed
Add drop view test (#253)

w41ter
authored and
Vallishp
committed
Fix test_cds_tbl_alter_replace (#254)

w41ter
authored and
Vallishp
committed
Fix partition dropped in partial snapshot (#255)

w41ter
authored and
Vallishp
committed
Add ts/ds test for tbl model (#244)

wyxxxcat
authored and
Vallishp
committed
Add ts/ds test for properties partition and bucket (#246)

wyxxxcat
authored and
Vallishp
committed
Organize index related suites (#256)

w41ter
authored and
Vallishp
committed
Add and drop [ng]bloom filter suites (#257)

w41ter
authored and
Vallishp
committed
Add bloom filter fpp suites (#258)

w41ter
authored and
Vallishp
committed
Add ts/ds test for table properties (#251)

wyxxxcat
authored and
Vallishp
committed
Support txn insert when table sync (#234)

lsy3993
authored and
Vallishp
committed
Add flag to controll txn insert (#259)

w41ter
authored and
Vallishp
committed
Fix problem caused by database name too long for test (#260)

wyxxxcat
authored and
Vallishp
committed
fix format (#261)

w41ter
authored and
Vallishp
committed
Fix create table if table already exists (#262)

w41ter
authored and
Vallishp
committed
Add docs for ccr regression test (#266)

wyxxxcat
authored and
Vallishp
committed
Refactor partial snapshot with replace/rename (#267)

w41ter
authored and
Vallishp
committed
Fix incorrect test (#265)

wyxxxcat
authored and
Vallishp
committed
Support rename partition/rollup (#268)

w41ter
authored and
Vallishp
committed
Support add/drop rollup binlogs (#269)

w41ter
authored and
Vallishp
committed
Fix modify view info/modify comment binlog in barrier log (#270)

w41ter
authored and
Vallishp
committed
Add rollup schema change cases (#272)

w41ter
authored and
Vallishp
committed
Fix alter view def (#273)

w41ter
authored and
Vallishp
committed
[improve] API job_progress shouldn't returns the data field of job progress (#271)

wyxxxcat
authored and
Vallishp
committed
Supplement table property compaction_policy test (#274)

wyxxxcat
authored and
Vallishp
committed
Fix restore with view (#275)

w41ter
authored and
Vallishp
committed
fix test_ds_mv_basic/test_ds_view_basic (#276)

w41ter
authored and
Vallishp
committed
Fix test_ds_view_basic (#277)

w41ter
authored and
Vallishp
committed
Support replace table in table sync (#279)

w41ter
authored and
Vallishp
committed
Update CHANGELOG.md (#281)

w41ter
authored and
Vallishp
committed
Retry handle upsert binlog, to fix meta error (#282)

w41ter
authored and
Vallishp
committed
Fix wrong infinity partition key in create table sql (#285)

w41ter
authored and
Vallishp
committed
Support create table with agg_state (#286)

w41ter
authored and
Vallishp
committed
Fix test_ds_tbl_res_agg_state (#287)

w41ter
authored and
Vallishp
committed
Support private IP and public IP (#288)

w41ter
authored and
Vallishp
committed
Update operations.md (#289)

w41ter
authored and
Vallishp
committed
Add alter table property for ds/ts test (#264)

wyxxxcat
authored and
Vallishp
committed
Fix spelling mistake (#291)

lsy3993
authored and
Vallishp
committed
Add db incremental sync test for table property (#280)

wyxxxcat
authored and
Vallishp
committed
Support add/drop multi inverted indexes (#296)

w41ter
authored and
Vallishp
committed
Fix fullsync commit seq with views (#297)

w41ter
authored and
Vallishp
committed
[TEST] fix error test for ccr (#300)

wyxxxcat
authored and
Vallishp
committed
Fix be rpc leaks in rpc factory (#299)

w41ter
authored and
Vallishp
committed
[feat](binlog) Add Support recover binlog (#284)
Vallishp
committed
Fix format (#302)

w41ter
authored and
Vallishp
committed
[Enhancement] add support for restore to ccr
Vallishp
committed