2.8.0

v2.8.0

NEW FEATURES

• add rebase command and POST /backup/rebase/{name} API endpoint — copy required parts from the required_backup chain into a remote incremental backup via server-side CopyObject (with streaming fallback) and remove the required_backup dependency, so the incremental backup becomes a full one without re-uploading data from the ClickHouse host; per-table parallelism is controlled by general.rebase_concurrency (env REBASE_CONCURRENCY, default = download_concurrency); requires backups made with upload_by_part: true and the same compression_format across the chain, fix #1344, #1444
• add general.rebase_before_remove_old_remote (env REBASE_BEFORE_REMOVE_OLD_REMOTE, default false) — makes backups_to_keep_remote a strict limit: when deletion of old remote backups is blocked by required_backup links from kept backups, the oldest kept increment is rebased first (same as the rebase command) so the whole out-of-window chain becomes deletable; rebase failure is not fatal and falls back to the legacy keep-required behavior
• add --schedule to watch command, general.watch_schedules config section (env WATCH_SCHEDULES, ;-separated) and the schedule query argument to POST /backup/watch — named cron-driven backup chains in name=<name>,full=<cron>[,increment=<cron>][,full_type=create|rebase][,delete_previous_cycle=true|false] format (standard 5-field cron, optional leading seconds field, @every/@daily descriptors), can be specified multiple times, mutually exclusive with --watch-interval/--full-interval; name is added as a prefix to watch_backup_name_template to isolate backup chains, full_type=rebase creates the scheduled full backup as increment + rebase (server-side copy of the previous chain instead of a full re-upload), delete_previous_cycle=true deletes all older backups of the chain after a successful full backup, fix #1354
• add --list-parts and --partitions (--list-partitions) flags to tables command — additionally list every physical part (name, partition_id, size) or the distinct partitions (partition_id, partition, parts count, size) for each table; against the live server the data comes from system.parts, against --local-backup/--remote-backup part names come from backup metadata (partition_id derived from the part name), fix #959, #1333
• add --rbac, --configs, --named-collections flags to server --watch — the server command never passed them through, so watch mode backed up only data even though the standalone watch command and the REST API endpoints already supported them, fix #955
• add rebalance command and POST /backup/rebalance/{name} API endpoint (with --tables and --dry-run support) — move data parts inside a local backup between disks: hardlink from the live part when system.parts shows it on another disk; parts absent from system.parts are copied to the storage-policy disk of the same type with the most free space (unreserved_space - keep_free_space from system.disks), parts on a disk which is absent or left the policy always move; parts on object disks are skipped, fix #1024
• add gcs.allow_multipart_upload (env GCS_ALLOW_MULTIPART_UPLOAD, default false, experimental) with gcs.upload_concurrency and gcs.multipart_upload_min_size (default 1GB) — files bigger than multipart_upload_min_size upload to GCS as multiple parallel parts (part size is chunk_size) via gRPC client and compose into the final object; not compatible with endpoint, force_http, encryption_key, fix #1028
• add gcs.allow_multipart_download (env GCS_ALLOW_MULTIPART_DOWNLOAD, default false) with gcs.download_concurrency — download each file as parallel range reads (part size is chunk_size) into a temporary file, mirrors s3.allow_multipart_download, fix #1028
• add general.disable_environment_override (settable ONLY in the config file, no environment variable name on purpose, default false) — when true, config values come only from the config file: all environment variables and the --env CLI flag are ignored during config loading; protects against accidental overrides such as Kubernetes service-discovery variables, fix #1079

IMPROVEMENTS

• write a gzip-compressed bbolt file manifest manifest.bolt.gz during upload and read it during download/restore to eliminate the per-part Walk (ListObjectsV2 and analogs) calls to remote storage that dominated restore time on backups with many parts; memory stays bounded even with millions of files (batched writes on upload, mmap-backed prefix seek on read), with graceful fallback to Walk when the manifest is missing or unreadable, and manifest failures are non-fatal for the backup, fix #1375, #1380
• store per-part size in table metadata during create and check the required free disk space right after table metadata download, before downloading any data — the check respects --tables/--partitions, resolves sizes of required parts through the required-backup chain, and when --hardlink-exists-files is used doesn't count parts which can be hardlinked from existing local parts (matched by hash_of_all_files via system.parts or by CRC64 of checksums.txt), fix #1268
• fail fast on canceled context in the retrier — SIGTERM / POST /backup/kill no longer trigger a useless extra retry attempt with a misleading Will wait and retry warning per goroutine; during API server shutdown log which in-flight command and cancel_operation_timeout the server is waiting for instead of blocking silently

BUG FIXES

• fix silent truncation of compressed backup uploads to S3 when s3.check_sum_algorithm: CRC32 (Object Lock buckets): the multipart upload stopped at the declared size (sum of raw file sizes), dropping the tar headers/padding/trailer tail of the archive; the stream is now consumed until EOF, fix #1471
• fix rbac_conflict_resolution: ignore during restore --rbac — RBAC objects from the backup that already exist on the server are now really ignored: their access/*.sql files are not copied over the live ones and the correlated replicated uuid znodes and name-to-uuid mappings are not restored into Keeper, fix #1013

2.7.4

v2.7.4

NEW FEATURES

• add clickhouse.parts_columns_max_bytes_before_external_group_by (env CLICKHOUSE_PARTS_COLUMNS_MAX_BYTES_BEFORE_EXTERNAL_GROUP_BY, default 100000000) and clickhouse.parts_columns_max_memory_usage (env CLICKHOUSE_PARTS_COLUMNS_MAX_MEMORY_USAGE, default 200000000) — make the memory caps injected into the system.parts_columns check query configurable instead of hardcoded, 0 means don't inject the corresponding setting (pre-2.7.2 behavior), fix #1420

BUG FIXES

• fix --restore-database-mapping/--restore-table-mapping losing unrelated comma-separated --tables pattern items, or applying the mapping to the wrong item when the same source database/table name was repeated across items — each comma-separated pattern item is now matched and substituted independently, fix #1421

2.7.3

v2.7.3

NEW FEATURES

• add general.max_broken_part_ratio (env MAX_BROKEN_PART_RATIO, default 0) — when >0, create/create_remote skips broken data parts instead of aborting the whole backup as long as the broken/total part ratio stays at or below the configured fraction; skipped parts are logged as warnings and the backup is marked successful but partial; 0 preserves the legacy any-broken-part-fails behavior, fix #1418
• add clickhouse.rebind_replica_path_if_exists (env CLICKHOUSE_REBIND_REPLICA_PATH_IF_EXISTS, CLI --rebind-replica-path-if-exists, also accepted by the REST API POST /backup/restore endpoint) — opt-in rebind of the replicated table ZK path to default_replica_path during restore when the resolved path already has leftover children but no local table uses it (async-stale state after DROP); MUST stay false during a concurrent multi-replica restore to avoid split-brain, fix #849
• add gcs.disable_http2 (env GCS_DISABLE_HTTP2) — force HTTP/1.1 for the GCS client to avoid single-TCP-connection HTTP/2 multiplexing becoming the bottleneck for high-throughput parallel downloads, fix #1437

IMPROVEMENTS

• switch the S3 client retry mode from aws.RetryModeStandard to aws.RetryModeAdaptive, so upload/download retries back off with a client-side rate limiter under throttling (SlowDown/503) instead of a fixed token bucket
• support tables with ENGINE=Alias during restore — strip the explicit column list (Alias inherits columns from the target table) and add allow_experimental_alias_table_engine=1 for ClickHouse 25.11+, fix #1426
• skip tables from databases with engine DataLakeCatalog (read-only external catalogs) during backup, in addition to MySQL/PostgreSQL/MaterializedPostgreSQL, fix #1417
• speed up create on servers with many tables: batch the in-progress system.mutations lookup instead of issuing one query per table (O(N^2) -> O(N)), fix #1431
• speed up restore/download table list resolution: eliminate the O(N^2) linear duplicate scan in getTableListByPatternLocal/getTableListByPatternRemote via a shared O(1) dedup keyed on metadata.TableTitle, fix #1430
• extend TestFlows FIPS 140-3 test coverage: dedicated test steps, stronger --fips-info/version-output and TLS-handshake assertions, updated requirements documentation, fix #1443

BUG FIXES

• restore the default general.compression_use_multi_thread to true: #1378 defaulted it to false, which silently dropped gzip/zstd compression from multi-threaded (pre-1378 gzip always used pgzip) to single-threaded and caused ~30% slower upload throughput for backups dominated by one large table (where upload_concurrency provides no per-stream parallelism); the option is now silently ignored instead of failing config validation for formats other than gzip/zstd
• fix v2.7.1 regression where restore --schema on a second replica rebound the ZK replica path even while the first replica was active and healthy — a foreign/renamed local table occupying the resolved ZK path is still detected automatically via system.replicas and rebound to default_replica_path, but a non-empty ZK path without a local table now defaults to the HA-safe join and requires the new opt-in rebind_replica_path_if_exists, fix #1428
• fix inconsistent --rbac behavior: restore --rbac no longer returns an error when the backup contains no RBAC access directory, matching the download --rbac behavior, fix #1432
• inject ON CLUSTER for LIVE VIEW and WINDOW VIEW restored with explicit column lists when restore_schema_on_cluster is set — enrichQueryWithOnCluster missed these statements and the views were created only on the local replica, fix #1433
• mask sensitive values (passwords, storage credentials, encryption keys) when logging --env config overrides, and in the API server action logs and /backup/status output, fix #1441

2.7.2

v2.7.2

IMPROVEMENTS

• include the data part file path and source object key (fPath, srcKey) in object_disk CopyObject/CopyObjectStreaming error messages during create and restore, so broken object_disk data (keys missing on remote storage) points to the exact failing file instead of just the shadow/table name
• document GCS Workload Identity authentication for the gcs remote storage in Examples.md

BUG FIXES

• fix clickhouse.skip_table_engines (env CLICKHOUSE_SKIP_TABLE_ENGINES) silently keeping some matching tables: the in-place slice removal advanced the cursor past the next element, so adjacent tables sharing a skipped engine were not all skipped; iterate in reverse so every match is dropped, fix #1416
• ensure /backup/kill (and context cancellation in general) promptly aborts an in-flight download/restore even when a read is stalled on a slow/half-open network or disk backpressure — the source reader is now force-closed on cancellation so blocked Read calls in DownloadCompressedStream/DownloadPath, the Azure CopyObject poll backoff, and object_disk.CopyObjectStreaming return instead of running to completion, and the resumable .pid file is removed, fix #1365

2.7.1

v2.7.1

NEW FEATURES

• add --fips-info app-level flag — prints binary name, version, git commit, build date, Go version, and the FIPS module build/runtime state (GOFIPS140 build setting, GODEBUG fips140 default/runtime) then exits, without requiring a Go toolchain, fix #1402
• add Azure AD Workload Identity support for azblob — when AZBLOB_USE_MANAGED_IDENTITY=true and the AZURE_TENANT_ID/AZURE_CLIENT_ID/AZURE_FEDERATED_TOKEN_FILE env vars are injected (e.g. by the AAD Workload Identity webhook), the federated token is used to authenticate; see Examples.md for deployment, fix #1124

IMPROVEMENTS

• add general.compression_use_multi_thread (env COMPRESSION_USE_MULTI_THREAD, default false), general.compression_threads (env COMPRESSION_THREADS, default 0 = auto/GOMAXPROCS) and general.compression_buffer_size (env COMPRESSION_BUFFER_SIZE, default 0) config options to tune per-stream zstd/gzip threading and the compression buffer (zstd encoder window / gzip DEFLATE window / pgzip block size); per-stream compression is now single-threaded by default to avoid CPU over-subscription, since upload_concurrency/download_concurrency already parallelize across tables, fix #1378
• migrate compression from the archived, frozen github.com/mholt/archiver/v4 v4.0.0-alpha.8 to its maintained successor github.com/mholt/archives v0.1.5, removing the replace directive pinned in archiver#428
• migrate S3 storage from the deprecated github.com/aws/aws-sdk-go-v2/feature/s3/manager to github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager; the forced CRC32 aws-chunked trailer that broke non-AWS S3-compatible providers is now disabled via RequestChecksumCalculation=WhenRequired, and the obsolete s3.buffer_size / S3_BUFFER_SIZE option was removed, fix #1409
• add buffer-size and HTTP-transport tuning for high-bandwidth (10Gbps+) S3/GCS transfers: general.pipe_buffer_size (env PIPE_BUFFER_SIZE, default 128KB), general.download_copy_buffer_size (env DOWNLOAD_COPY_BUFFER_SIZE), gcs.upload_buffer_size (env GCS_UPLOAD_BUFFER_SIZE), s3.http_write_buffer_size (env S3_HTTP_WRITE_BUFFER_SIZE) and s3.http_read_buffer_size (env S3_HTTP_READ_BUFFER_SIZE), fix #1376
• replace post-hoc sleep-based bandwidth throttling with a token-bucket rate limiter wrapped around the storage Reader/Writer interfaces, so the configured limit is enforced continuously during transfer instead of after each chunk, fix #934, #1377
• parallelize ALTER TABLE ... UNFREEZE after backup create instead of running it inline inside each table goroutine, so an UNFREEZE no longer holds an upload_concurrency slot and blocks the next table, fix #1381
• add clickhouse.parts_columns_batch_size (env CLICKHOUSE_PARTS_COLUMNS_BATCH_SIZE, default 25) to batch the system.parts lookups when computing hash_of_all_files, avoiding Max query size exceeded failures on tables with very many parts, fix #1408
• resolve required data parts during restore the same way as during download — hardlink from the required backup on local disk when present, otherwise download the part from remote storage, fix #1023
• ResumeOperationsAfterRestart now ignores create.state2/restore.state2 files instead of failing the API server startup with unknown command; only upload and download are auto-resumed after a server restart, fix #1083
• fail fast with a clear error instead of retrying for ~35s when a remote table metadata .json is missing during download (covers S3 NoSuchKey, GCS 404, Azure BlobNotFound, FTP/SFTP not-found) — a missing table file is a permanent broken-backup condition, not a transient one, fix #1379
• emit a clear error on --resume download when the local backup exists but download.state2 is missing (so it is unknown which parts are complete), instead of crashing or silently resuming on top of partial data, fix #1383
• harden FIPS 140-3 verification: native Go GOFIPS140=v1.0.0 checks, ACVP reproducibility tests, outbound S3 TLS rejection checks and container cleanup in CI/CD, fix #1399, #1401, #1404

BUG FIXES

• don't kill clickhouse-backup server with Fatal/os.Exit when the resumable state DB can't be written or read (e.g. no space left on device); the error now propagates so the server stays alive and returns it to the API client, while the CLI exits with a non-zero code, fix #1172
• fix backup create failing with part "<name>" not found in system.parts ... after FREEZE when a ClickHouse cache disk (e.g. s3_cache) wraps an underlying S3 object disk — prefer the underlying disk name over the cache wrapper in getDisksFromSystemDisks, fix #1396
• fix --hardlink-exists-files to also match parts whose hash_of_all_files is identical but that now live under a renamed table, fix #1398
• fix --table combined with --resume on incremental backups: recursively downloading a required backup closed the parent b.dst connection and wiped the resumable state; the connection is now saved/restored, fix #1384
• improve detection when clickhouse-backup runs on a host whose disks differ from clickhouse-server, instead of silently warning doesn't contain tables for restore, fix #1037

2.7.0

v2.7.0

NEW FEATURES

• add clean_broken_retention CLI command — walks top-level of remote path and object_disks_path and batch-deletes (with retry) every entry that is not present in the live backup list and not matched by any --exclude=<glob> (and optionally scoped by --include=<glob>). Dry-run by default; pass --commit to actually delete. Useful for cleaning up orphans left by failed retention runs, fix #1371
• add info CLI command for per-table backup size breakdown — shows per-table size, part count, and disk breakdown for local and remote backups, supports --tables=<db>.<table> glob filter and --format=text|json|yaml|csv|tsv, accepts all|local|remote scope, fix #1388
• add force_rebalance config option (clickhouse.force_rebalance, env CLICKHOUSE_FORCE_REBALANCE) — distribute restored data across multiple JBOD disks under the same storage policy even when the source disk name (e.g. default) exists on the target machine, fix #1350
• switch FIPS variant from FIPS 140-2 boringssl to native Go 1.24+ FIPS 140-3 (GODEBUG=fips140=on); embed an ACVP wrapper into the shipped clickhouse-backup-fips binary with dual entry points (clickhouse-backup-acvp argv0 dispatch and clickhouse-backup acvp subcommand) and ship a tracked public-scope ACVP reproducibility flow, fix #1341, #1364, #1391, #1395
• add safety check to restore/restore_remote: fail without --rm/--drop when target tables already exist and contain rows (checked via clusterAllReplicas('{cluster}') when restore_schema_on_cluster is set) to avoid dangerous accidental DROP TABLE, fix #1325
• add checksum verification during upload --diff-from / --diff-from-remote when part name matches, to avoid uploading mismatched data and to detect silent corruption, fix #1307

IMPROVEMENTS

• speed up restore_remote from S3 incremental chains: cache backup list and avoid redundant ListObjects calls per table (previously 8h on 280GB / 3500 tables shrinks to minutes), fix #1362, #1361
• reduce backup memory footprint for databases with thousands of tables (regression introduced in v2.6.42), fix #1360
• wrap S3 credentials with aws.NewCredentialsCache() to avoid resolving credentials on every API call (IMDS/STS), reducing latency and throttling in IRSA + AssumeRole flows, fix #1335
• simplify hash_of_all_files computation via a single post-FREEZE SELECT from system.parts instead of per-file hashing — also enables --hardlinks-exists-files to consult system.parts checksums during download, fix #1338
• isolate FREEZE shadow directory per backup as /var/lib/clickhouse/shadow/backup-{uuid} so concurrent backups and cleanup-after-failure don't clobber each other's shadow data, fix #1345
• add option to skip persisting list calls into the API server actions state — prevents unbounded growth of actions state when /backup/list is used as a monitoring endpoint during long-running backups, fix #1359
• improve kill command to ensure all in-flight operations really finish and to remove leftover .pid files, fix #1365
• document missing/incorrect concurrency defaults in ReadMe.md (download_concurrency, s3.concurrency, cos.concurrency, sftp.concurrency, ftp.concurrency), fix #1346
• migrate integration tests to testcontainers-go for better parallelism and isolation, fix #1336
• fix the list_duration log field formatting in pkg/storage/general.go (was emitting raw nanoseconds), fix #1337

BUG FIXES

• fix restore_remote for tables using sparse-column serialization: accept empty sparse metadata files instead of treating StorageObjectCount=0 as corruption, affects ClickHouse 23.8+, fix #1372
• fix restore_remote aborting the entire restore when an incremental backup contains a table absent from the required full backup; the missing table is now skipped with a warning, fix #1373
• fix object_disk backup on S3 sources with SSE-C: handle 404 from server-side CopyObject by falling back to streaming and stop issuing HeadObject (returns 400) on SSE-C source objects before GetObject, fix #1374
• fix --rbac-only backup failing with "is empty backup" when the database contains RBAC objects but no tables and allow_empty_backups=false, fix #1355
• fix nested ssh consuming stdin from the while read loop in the rsync helper (use ssh -n) so all backup metadata files are processed instead of only the first, fix #1368
• fix backup retention logic in the rsync helper: correct line counting, numeric comparison and arithmetic handling so old backups are properly cleaned up, fix #1369

2.6.44

v2.6.44

BUG FIXES

• fix 1356, retry batch deletion in cleanBackupObjectDisks so transient errors (e.g. GCS 503) during retention no longer leave orphaned objects in object_disks_path

2.6.43

v2.6.43

NEW FEATURES

• add S3_REQUEST_CONTENT_MD5 option for S3-compatible storage backends that require Content-MD5 header on uploads (e.g. Huawei S3), fix 1324, fix 1329

IMPROVEMENTS

• add ClickHouse 26.1, 26.2 to test matrix
• refactor integration tests: split monolithic test file and reorganize configs for better maintainability

BUG FIXES

• fix GCS upload performance regression by letting SDK manage transport instead of custom HTTP client
• fix restore --partitions option error related to Embedded backup
• fix 1328, restore --data --partitions now correctly uses DROP PARTITION ID '<id>' for partition IDs and DROP PARTITION (<tuple>) for tuple-format partitions instead of always using DROP PARTITION, also fix the same issue for Embedded backup/restore

2.6.42

v2.6.42

NEW FEATURES

• apply object disk key rewriting for remapped tables during restore to prevent data corruption when using --restore-database-mapping or --restore-table-mapping with object disks (S3/GCS/Azure), fix 1278
• add GCS customer-supplied encryption key (CSEK) support for client-side encryption where the encryption key is controlled by the user, not Google. Use GCS_ENCRYPTION_KEY environment variable or gcs.encryption_key config option with base64-encoded 256-bit key, fix 1316
• add TLS support for Keeper connections, allows secure connections to ClickHouse Keeper with SSL/TLS certificates, fix 1312
• add --skip-empty-tables option to restore and restore_remote commands to skip tables with no data during restore, available in CLI, API handlers, and server mode, fix 1265

IMPROVEMENTS

• implements batching keys deletions to improve speed of delete old backups during backup retention, fix 1066
• add ClickHouse 25.12 to test matrix
• add example for minimal grants for backup user in Examples.md
• improve GCS connection handling: properly close readers on error to prevent goroutine leaks, change retry logging from Debug to Warn level
• use partitionId directly instead of INSERT INTO temp table for ClickHouse 21.8+, improves partition handling performance, fix 1315
• refactor table column type checking to use single query before freeze operation instead of per-table queries fix 1194
• add KEEPER_TLS_ENABLES=1 by default in integration tests
• improve TestKeeperTLS, TestReplicatedCopyToDetached, and TestRestoreDistributedCluster test stability
• update GitHub Actions workflows and GOROOT configuration
• explain S3_FORCE_PATH_STYLE configuration option in documentation

BUG FIXES

• fix S3 multipart operations (upload, download, copy) to respect S3_MAX_PARTS_COUNT instead of hardcoded 10000 value, allows S3-compatible backends with stricter limits
• fix GCS credential conflicts when multiple authentication options are provided, refactor GCS Connect to avoid conflicting client options
• fix COS upload for large files, fix 1318
• fix list status update in server mode, fix 1317
• fix TestServerAPI and TestReplicatedCopyToDetached test failures in CI/CD
• fix OpenSSL/client config parsing for Keeper TLS connections, add comprehensive integration tests
• fix partition filtering when using --restore-database-mapping, --restore-table-mapping together with --partitions option
• security: update dependencies to fix CVE-2025-61729, CVE-2025-61727

2.6.41

v2.6.41

BUG FIXES

• improve restore long RBAC which have length more 64k, fix 1305