Skip to content

Commit 87036f8

Browse files
dr-mdr-m
committed
MDEV-14992 BACKUP SERVER
This introduces a basic driver Sql_cmd_backup, storage engine interfaces, and basic copying of InnoDB, Aria, and MyISAM files. For streaming, we aim to generate streams that are compatible with GNU tar --format=oldgnu, which is also supported by the built-in BSD tar on FreeBSD, macOS as well as Microsoft Windows. That is, to extract files from a backup stream, you can use the standard tar utility of the operating system, instead of anything nonstandard like xbstream or mbstream. TODO: Support partial backup and restore. This should be done by implementing some configuration parameters as well as a predicate that checks if a file name pattern should be included. TODO: Back up ENGINE=RocksDB. TODO: Refactor the crude first implementation of Aria_backup that is based on the work of Andrzej Jarzabek, to copy as many files as concurrently as possible while holding the minimum amount of locks. --- backup_target: A structured data type to represent a target directory. On Microsoft Windows, we must use directory paths because there is no variant of CopyFileEx() that would work on file handles. backup_sink: Wraps a per-thread output stream as well as storage engine specific context. handlerton::backup_start(), handlerton::backup_end(): Invoked at the start or end of a backup phase, in the thread that executes a BACKUP SERVER statement. handlerton::backup_step(): A backup step that can be invoked from multiple threads concurrently, between the execution of the corresponding handlerton::backup_start() and handlerton::backup_end() of the same phase. copy_entire_file(): A file copying service for POSIX systems. copy_file(): A sparse file-copying service for all systems. backup_stream_append_async(): A variant of backup_stream_append() where the source file region is guaranteed to be immutable after the call returns. We must not use Linux sendfile(2) for copying data files that may be modified in place, because it could introduce a race condition between a page write that runs concurrently with a child process that is reading the data from the pipe. InnoDB_backup::context: Backup context, attached to backup_sink so that context can continue to exist between the time a BACKUP SERVER releases all locks and another BACKUP SERVER starts executing, with innodb_backup pointing to the new backup, while the old backup is still being finished. fil_space_t::write_or_backup: Keep track of in-flight page writes and pending backup operation. We must not allow them concurrently, because that could lead into torn pages in the backup. fil_space_t::backup_end: The first page number that is not being backed up (by default 0, to indicate that no backup is in progress). fil_space_t::BACKUP_BATCH_SIZE: The number of preceding pages that will be covered by fil_space_t::backup_end. This is the unit of "page range locking" during InnoDB backup. log_sys.backup: Whether BACKUP SERVER is in progress. The purpose of this is to make BACKUP SERVER prevent the concurrent execution of SET GLOBAL innodb_log_archive=OFF or SET GLOBAL innodb_log_file_size when innodb_log_archive=OFF. log_sys.archived_checkpoint: Keep track of the earliest available checkpoint, corresponding to log_sys.archived_lsn. This reflects SET GLOBAL innodb_log_recovery_start (which is settable now), for incremental backup. buf_flush_list_space(): Check for concurrent backup before writing each page. This is inefficient, but this function may be invoked from multiple threads concurrently, and it cannot be changed easily, especially for fil_crypt_thread().
1 parent eb0f328 commit 87036f8

92 files changed

Lines changed: 3645 additions & 124 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎libmysqld/CMakeLists.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -166,6 +166,7 @@ SET(SQL_EMBEDDED_SOURCES emb_qcache.cc libmysqld.c lib_sql.cc
166166
../sql/opt_hints.cc ../sql/opt_hints.h
167167
../sql/opt_trace_ddl_info.cc ../sql/opt_trace_ddl_info.h
168168
../sql/sql_path.cc
169+
../sql/sql_backup.cc
169170
${GEN_SOURCES}
170171
${MYSYS_LIBWRAP_SOURCE}
171172
)

‎mysql-test/collections/buildbot_suites.bat‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ innodb,^
66
versioning,^
77
plugins,^
88
mariabackup,^
9+
backup,^
910
roles,^
1011
auth_gssapi,^
1112
mysql_sha2,^

‎mysql-test/main/backup_server.result‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
BACKUP SERVER TO '$datadir/some_directory';
2+
ERROR HY000: Incorrect arguments to BACKUP SERVER TO
3+
BACKUP SERVER TO '$MYSQLTEST_VARDIR/some_directory';
4+
BACKUP SERVER TO '$MYSQLTEST_VARDIR/some_directory';
5+
ERROR HY000: Can't create directory 'MYSQLTEST_VARDIR/some_directory' (Errcode: 17 "File exists")
6+
BACKUP SERVER TO '$MYSQLTEST_VARDIR/some_directory';

‎mysql-test/main/backup_server.test‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
--let $datadir=`select @@datadir`
2+
--error ER_WRONG_ARGUMENTS
3+
evalp BACKUP SERVER TO '$datadir/some_directory';
4+
evalp BACKUP SERVER TO '$MYSQLTEST_VARDIR/some_directory';
5+
--replace_result $MYSQLTEST_VARDIR MYSQLTEST_VARDIR
6+
--error 21
7+
evalp BACKUP SERVER TO '$MYSQLTEST_VARDIR/some_directory';
8+
--rmdir $MYSQLTEST_VARDIR/some_directory
9+
evalp BACKUP SERVER TO '$MYSQLTEST_VARDIR/some_directory';
10+
--rmdir $MYSQLTEST_VARDIR/some_directory

‎mysql-test/main/backup_server_locking.result‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
BACKUP SERVER TO '$MYSQLTEST_VARDIR/some_directory';
2+
ERROR HY000: Can't create directory 'MYSQLTEST_VARDIR/some_directory' (Errcode: 17 "File exists")
3+
BACKUP STAGE START;
4+
connect backup,localhost,root;
5+
SET STATEMENT max_statement_time=0.1 FOR
6+
BACKUP SERVER TO '$MYSQLTEST_VARDIR/some_directory';
7+
ERROR 70100: Query was interrupted: execution time limit 0.1 sec exceeded
8+
connection default;
9+
BACKUP SERVER TO '$MYSQLTEST_VARDIR/some_directory';
10+
ERROR HY000: Can't execute the command as you have a BACKUP STAGE active
11+
BACKUP STAGE END;
12+
connection backup;
13+
SET STATEMENT max_statement_time=0.1 FOR
14+
BACKUP SERVER TO '$MYSQLTEST_VARDIR/some_directory';
15+
ERROR HY000: Can't create directory 'MYSQLTEST_VARDIR/some_directory' (Errcode: 17 "File exists")
16+
disconnect backup;
17+
connection default;

‎mysql-test/main/backup_server_locking.test‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
--source include/not_embedded.inc
2+
--source include/count_sessions.inc
3+
4+
--mkdir $MYSQLTEST_VARDIR/some_directory
5+
--replace_result $MYSQLTEST_VARDIR MYSQLTEST_VARDIR
6+
--error 21
7+
evalp BACKUP SERVER TO '$MYSQLTEST_VARDIR/some_directory';
8+
9+
BACKUP STAGE START;
10+
--connect (backup,localhost,root)
11+
--error ER_STATEMENT_TIMEOUT
12+
evalp SET STATEMENT max_statement_time=0.1 FOR
13+
BACKUP SERVER TO '$MYSQLTEST_VARDIR/some_directory';
14+
15+
--connection default
16+
17+
--error ER_BACKUP_LOCK_IS_ACTIVE
18+
evalp BACKUP SERVER TO '$MYSQLTEST_VARDIR/some_directory';
19+
20+
BACKUP STAGE END;
21+
--connection backup
22+
--replace_result $MYSQLTEST_VARDIR MYSQLTEST_VARDIR
23+
--error 21
24+
evalp SET STATEMENT max_statement_time=0.1 FOR
25+
BACKUP SERVER TO '$MYSQLTEST_VARDIR/some_directory';
26+
--disconnect backup
27+
--connection default
28+
29+
--rmdir $MYSQLTEST_VARDIR/some_directory
30+
31+
--source include/wait_until_count_sessions.inc

‎mysql-test/main/grant_backup_server.result‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
CREATE USER user1@localhost IDENTIFIED BY '';
2+
connect con1,localhost,user1;
3+
BACKUP SERVER TO 'some_directory';
4+
ERROR 42000: Access denied; you need (at least one of) the RELOAD privilege(s) for this operation
5+
disconnect con1;
6+
connection default;
7+
GRANT SELECT ON test.* TO user1@localhost;
8+
connect con1,localhost,user1;
9+
BACKUP SERVER TO 'some_directory';
10+
ERROR 42000: Access denied; you need (at least one of) the RELOAD privilege(s) for this operation
11+
disconnect con1;
12+
connection default;
13+
GRANT RELOAD ON test.* TO user1@localhost;
14+
ERROR HY000: Incorrect usage of DB GRANT and GLOBAL PRIVILEGES
15+
GRANT RELOAD ON *.* TO user1@localhost;
16+
connect con1,localhost,user1;
17+
BACKUP SERVER TO 'some_directory';
18+
ERROR 42000: Access denied; you need (at least one of) the SELECT privilege(s) for this operation
19+
disconnect con1;
20+
connection default;
21+
GRANT SELECT ON *.* TO user1@localhost;
22+
connect con1,localhost,user1;
23+
BACKUP SERVER TO '$datadir/some_directory';
24+
ERROR HY000: Incorrect arguments to BACKUP SERVER TO
25+
disconnect con1;
26+
connection default;
27+
DROP USER user1@localhost;

‎mysql-test/main/grant_backup_server.test‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
--source include/not_embedded.inc
2+
CREATE USER user1@localhost IDENTIFIED BY '';
3+
--connect con1,localhost,user1
4+
--error ER_SPECIFIC_ACCESS_DENIED_ERROR
5+
BACKUP SERVER TO 'some_directory';
6+
--disconnect con1
7+
--connection default
8+
GRANT SELECT ON test.* TO user1@localhost;
9+
--connect con1,localhost,user1
10+
--error ER_SPECIFIC_ACCESS_DENIED_ERROR
11+
BACKUP SERVER TO 'some_directory';
12+
--disconnect con1
13+
--connection default
14+
--error ER_WRONG_USAGE
15+
GRANT RELOAD ON test.* TO user1@localhost;
16+
GRANT RELOAD ON *.* TO user1@localhost;
17+
--connect con1,localhost,user1
18+
--error ER_SPECIFIC_ACCESS_DENIED_ERROR
19+
BACKUP SERVER TO 'some_directory';
20+
--disconnect con1
21+
--connection default
22+
GRANT SELECT ON *.* TO user1@localhost;
23+
--connect con1,localhost,user1
24+
--let $datadir=`select @@datadir`
25+
--error ER_WRONG_ARGUMENTS
26+
evalp BACKUP SERVER TO '$datadir/some_directory';
27+
--disconnect con1
28+
--connection default
29+
DROP USER user1@localhost;

‎mysql-test/main/mysqld--help.result‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2040,7 +2040,7 @@ performance-schema-max-socket-classes 10
20402040
performance-schema-max-socket-instances -1
20412041
performance-schema-max-sql-text-length 1024
20422042
performance-schema-max-stage-classes 170
2043-
performance-schema-max-statement-classes 227
2043+
performance-schema-max-statement-classes 228
20442044
performance-schema-max-statement-stack 10
20452045
performance-schema-max-table-handles -1
20462046
performance-schema-max-table-instances -1

‎mysql-test/mariadb-test-run.pl‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -180,6 +180,7 @@ END
180180
main-
181181
archive-
182182
atomic-
183+
backup-
183184
binlog-
184185
binlog_encryption-
185186
binlog_in_engine-

0 commit comments

Comments
 (0)