Merge branch 'master' into fips_140_autotests

Author: Slach

.github/workflows/build.yaml

@@ -340,10 +340,10 @@ jobs:
 
       - name: Running integration tests
         env:
-          RUN_PARALLEL: 4
           GOROOT: ${{ env.GOROOT_1_26_X64 }}
           CLICKHOUSE_VERSION: ${{ matrix.clickhouse }}
-          # LOG_LEVEL: "info"
+          RUN_PARALLEL: 4
+          # LOG_LEVEL: "debug"
           # TEST_LOG_LEVEL: "debug"
           # options for advanced debug CI/CD
           # RUN_TESTS: "^TestSkipDisk$"
@@ -369,6 +369,7 @@ jobs:
           QA_GCS_OVER_S3_BUCKET: ${{ secrets.QA_GCS_OVER_S3_BUCKET }}
         run: |
           set -xe
+          echo "RUN_PARALLEL=${RUN_PARALLEL}"
           echo "CLICKHOUSE_VERSION=${CLICKHOUSE_VERSION}"
           echo "GCS_TESTS=${GCS_TESTS}"
           GCS_ENCRYPTION_KEY=$(openssl rand -base64 32)
@@ -386,7 +387,7 @@ jobs:
           export CLICKHOUSE_BACKUP_BIN="$(pwd)/clickhouse-backup/clickhouse-backup-race"
 
           set +e
-          go test -count=1 -parallel "${RUN_PARALLEL}" -race -timeout "${TEST_TIMEOUT:-120m}" -failfast -tags=integration -shuffle=$(date +%Y%m%d) -run "${RUN_TESTS:-.+}" -v "./test/integration/..."
+          go test -count=1 -parallel "${RUN_PARALLEL}" -race -timeout "${TEST_TIMEOUT:-180m}" -failfast -tags=integration -shuffle=$(date +%Y%m%d) -run "${RUN_TESTS:-.+}" -v "./test/integration/..."
           TEST_FAILED=$?
           set -e
           if [[ "0" != "${TEST_FAILED}" ]]; then

ChangeLog.md

@@ -1,3 +1,8 @@
+# v2.7.1
+- add `general.compression_use_multi_thread` (env `COMPRESSION_USE_MULTI_THREAD`, default `false`), `general.compression_threads` (env `COMPRESSION_THREADS`, default `0` = auto/GOMAXPROCS) and `general.compression_buffer_size` (env `COMPRESSION_BUFFER_SIZE`, default `0`) config options to tune per-stream zstd/gzip threading and the compression buffer (zstd encoder window / gzip DEFLATE window / pgzip block size); per-stream compression is now single-threaded by default to avoid CPU over-subscription, since `upload_concurrency`/`download_concurrency` already parallelize across tables, fix [#1378](https://github.com/Altinity/clickhouse-backup/issues/1378)
+- migrate compression from the archived, frozen `github.com/mholt/archiver/v4 v4.0.0-alpha.8` to its maintained successor `github.com/mholt/archives v0.1.5`, removing the `replace` directive pinned in [archiver#428](https://github.com/mholt/archiver/issues/428)
+- don't kill `clickhouse-backup server` with `Fatal`/`os.Exit` when the resumable state DB can't be written or read (e.g. `no space left on device`); the error now propagates so the server stays alive and returns it to the API client, while the CLI exits with a non-zero code, fix [#1172](https://github.com/Altinity/clickhouse-backup/issues/1172)
+
 # v2.7.0
 
 NEW FEATURES

Examples.md

@@ -200,6 +200,36 @@ Notes for the other backends:
 - **cos**: the SDK uses `cos.DNSScatterTransport` for internal Tencent endpoints, which already spreads connections across IPs; raise `concurrency` rather than touching the HTTP pool.
 - **ftp**: the FTP library has no transfer-buffer knob; throughput is governed by `concurrency` (connection pool size).
 
+## Multi-threaded zstd/gzip compression
+
+By default `compression_use_multi_thread: false`, each compression stream is single-threaded. clickhouse-backup already
+parallelizes compression across tables via `upload_concurrency`/`download_concurrency`, so per-stream multi-threading
+mainly over-subscribes the CPU and reduces total throughput, see https://github.com/Altinity/clickhouse-backup/issues/1378.
+
+```yaml
+general:
+  remote_storage: s3
+  compression_use_multi_thread: true # COMPRESSION_USE_MULTI_THREAD, zstd WithEncoderConcurrency/WithDecoderConcurrency, gzip via pgzip
+  compression_threads: 8             # COMPRESSION_THREADS, per-stream threads (zstd concurrency / pgzip block workers); 0 = auto (GOMAXPROCS)
+  compression_buffer_size: 4194304   # COMPRESSION_BUFFER_SIZE, 4MB; zstd encoder window / pgzip block size (multi-threaded gzip)
+s3:
+  compression_format: zstd           # or gzip; the three settings above only affect zstd and gzip
+  compression_level: 3
+```
+
+Notes:
+- `compression_use_multi_thread` enables parallel zstd encode/decode (`WithEncoderConcurrency`/`WithDecoderConcurrency`)
+  and switches gzip to the parallel `pgzip` implementation.
+- `compression_threads` sets how many threads each stream uses when `compression_use_multi_thread` is enabled (zstd
+  concurrency / pgzip block workers); `0` means auto (`GOMAXPROCS`). It must be left at `0` when multi-thread is off.
+- `compression_buffer_size` meaning depends on the format and on `compression_use_multi_thread`:
+  - **zstd**: encoder window size (`WithWindowSize`), must be a power of two between 1024 and 536870912; larger windows
+    improve the compression ratio at the cost of more memory and CPU.
+  - **gzip, single-threaded**: DEFLATE window size (`gzip.NewWriterWindow`), 32..32768 — the gzip format caps the window
+    at 32KB, so values above that are rejected.
+  - **gzip, multi-threaded**: `pgzip` block size (`SetConcurrency`), must be greater than 16384 (e.g. 1–4MB).
+- The other formats (`lz4`, `bzip2`, `sz`, `xz`, `brotli`) ignore both settings.
+
 ## How to use clickhouse-backup in Kubernetes
 
 Install the [clickhouse kubernetes operator](https://github.com/Altinity/clickhouse-operator/) and use the following

ReadMe.md

@@ -114,13 +114,21 @@ general:
   download_concurrency: 1        # DOWNLOAD_CONCURRENCY, max 255, by default, the value is floor(AVAILABLE_CPU_CORES / 2). If result is < 1, then 1.
   upload_concurrency: 1          # UPLOAD_CONCURRENCY, max 255, by default, the value is round(sqrt(AVAILABLE_CPU_CORES / 2)). If result is < 1, then 1.
 
-  # Throttling speed for upload and download, calculates on part level, not the socket level, it means short period for high traffic values and then time to sleep 
+  # Throttling speed for upload and download, enforced inline via a token-bucket rate limiter shared across all concurrent workers, so the value is an aggregate cap with smooth (non-bursty) throughput.
+  # Throttling does NOT apply to server-side object disk copy (`CopyObject`, e.g. S3 server-side copy / Azure copy-blob), because those bytes move inside the cloud provider and never pass through clickhouse-backup. When server-side copy is unavailable (incompatible src/dst `remote_storage`, or a failed `CopyObject` falling back to streaming through local memory), the streaming copy IS throttled.
+  # Throttling does NOT apply to `remote_storage: custom`, because data transfer happens inside your external upload/download command, not through clickhouse-backup.
+  # When `clickhouse->use_embedded_backup_restore: true`, throttling is delegated to the ClickHouse server via the `max_backup_bandwidth` query setting passed in the BACKUP/RESTORE SETTINGS clause (requires ClickHouse 25.1+); upload_max_bytes_per_second applies to BACKUP, download_max_bytes_per_second to RESTORE. On older ClickHouse versions embedded transfers are not throttled.
   download_max_bytes_per_second: 0  # DOWNLOAD_MAX_BYTES_PER_SECOND, 0 means no throttling 
   upload_max_bytes_per_second: 0    # UPLOAD_MAX_BYTES_PER_SECOND, 0 means no throttling
 
   # Buffer tuning for high-bandwidth (10Gbit+) networks, see https://github.com/Altinity/clickhouse-backup/issues/1376 and Examples.md#tuning-for-high-bandwidth-10gbit-networks
   pipe_buffer_size: 131072          # PIPE_BUFFER_SIZE, size in bytes of the in-memory ring buffer between the compression and the upload/download stream handlers, default 128KB; raise (e.g. 8388608 = 8MB) to let compression run ahead of uploads on fast networks
   download_copy_buffer_size: 0      # DOWNLOAD_COPY_BUFFER_SIZE, explicit buffer size in bytes for io.CopyBuffer during download/extract, 0 means use the Go default (32KB); raise (e.g. 1048576 = 1MB) to reduce syscalls per file on fast networks
+
+  # zstd/gzip compression tuning, see https://github.com/Altinity/clickhouse-backup/issues/1378 and Examples.md#multi-threaded-zstdgzip-compression
+  compression_use_multi_thread: false # COMPRESSION_USE_MULTI_THREAD, enable per-stream multi-threaded zstd/gzip compression and decompression; default false because upload_concurrency/download_concurrency already parallelize across tables, so per-stream threading mainly over-subscribes CPU. Enable when backing up a single large table with low concurrency. Only affects compression_format: zstd and gzip
+  compression_threads: 0            # COMPRESSION_THREADS, number of per-stream compression threads when compression_use_multi_thread is enabled (zstd concurrency / pgzip block workers), 0 means auto (GOMAXPROCS); must be unset/0 when compression_use_multi_thread is false
+  compression_buffer_size: 0        # COMPRESSION_BUFFER_SIZE, compression buffer size in bytes, 0 keeps library defaults. Meaning and valid range depend on compression_format and compression_use_multi_thread: zstd = encoder window (power of two, 1024..536870912, e.g. 4194304 = 4MB); single-threaded gzip = DEFLATE window (32..32768); multi-threaded gzip = pgzip block size (>16384). Other formats reject it
 
   # when table data contains in system.disks with type=ObjectStorage, then we need execute remote copy object in object storage service provider, this parameter can restrict how many files will copied in parallel  for each table 
   object_disk_server_side_copy_concurrency: 32
@@ -230,6 +238,7 @@ clickhouse:
   restore_as_attach: false # CLICKHOUSE_RESTORE_AS_ATTACH, allow restore tables which have inconsistent data parts structure and mutations in progress
   restore_distributed_cluster: "" # CLICKHOUSE_RESTORE_DISTRIBUTED_CLUSTER, cluster name (can use macros) which will use during restore `engine=Distributed` tables, when cluster defined in backup table definition not exists in `system.clusters`
   check_parts_columns: true # CLICKHOUSE_CHECK_PARTS_COLUMNS, check data types from system.parts_columns during create backup to guarantee mutation is complete
+  parts_columns_batch_size: 25 # CLICKHOUSE_PARTS_COLUMNS_BATCH_SIZE, batch size for system.parts_columns checks
   max_connections: 0 # CLICKHOUSE_MAX_CONNECTIONS, how many parallel connections could be opened during operations
 azblob:
   endpoint_schema: "https"            # AZBLOB_ENDPOINT_SCHEMA, URL scheme used to build the AZBLOB endpoint (e.g. http for Azurite emulator)
@@ -291,7 +300,6 @@ s3:
   delete_concurrency: 10           # S3_DELETE_CONCURRENCY, how many parallel DeleteObjects requests during clean/delete operations
 
   # HTTP transport and buffer tuning for high-bandwidth (10Gbit+) networks, see https://github.com/Altinity/clickhouse-backup/issues/1376 and Examples.md#tuning-for-high-bandwidth-10gbit-networks
-  buffer_size: 65536                  # S3_BUFFER_SIZE, per-part buffer in bytes for the s3manager up/downloader, default 64KB; raise (e.g. 1048576 = 1MB) on fast networks
   http_max_idle_conns: 0              # S3_HTTP_MAX_IDLE_CONNS, http.Transport.MaxIdleConns, 0 keeps the AWS SDK default
   http_max_idle_conns_per_host: 0     # S3_HTTP_MAX_IDLE_CONNS_PER_HOST, http.Transport.MaxIdleConnsPerHost, 0 keeps the Go default (2); raise (e.g. 128) to avoid serializing parallel up/downloads to the same endpoint when concurrency is high
   http_max_conns_per_host: 0          # S3_HTTP_MAX_CONNS_PER_HOST, http.Transport.MaxConnsPerHost, 0 means unlimited
@@ -400,7 +408,8 @@ api:
   integration_tables_host: ""  # API_INTEGRATION_TABLES_HOST, allow using DNS name to connect in `system.backup_list` and `system.backup_actions`
   allow_parallel: false        # API_ALLOW_PARALLEL, enable parallel operations, this allows for significant memory allocation and spawns go-routines, don't enable it if you are not sure
   create_integration_tables: false # API_CREATE_INTEGRATION_TABLES, create `system.backup_list` and `system.backup_actions`
-  complete_resumable_after_restart: true # API_COMPLETE_RESUMABLE_AFTER_RESTART, after API server startup, if `/var/lib/clickhouse/backup/*/(upload|download).state2` present, then operation will continue in the background
+  complete_resumable_after_restart: true # API_COMPLETE_RESUMABLE_AFTER_RESTART, after API server startup, if `/var/lib/clickhouse/backup/*/{command}.state2` present and command is allowed by complete_resumable_after_restart_commands, then operation will continue in the background
+  complete_resumable_after_restart_commands: [upload, download] # API_COMPLETE_RESUMABLE_AFTER_RESTART_COMMANDS, commands allowed for automatic resume after API server restart
   watch_is_main_process: false # WATCH_IS_MAIN_PROCESS, treats 'watch' command as a main api process, if it is stopped unexpectedly, api server is also stopped. Does not stop api server if 'watch' command canceled by the user. 
   backup_actions_skip_commands: [] # API_BACKUP_ACTIONS_SKIP_COMMANDS, list of commands that must NOT be recorded into the in-memory async status exposed via `system.backup_actions` and `/backup/actions`. Useful to keep high-frequency monitoring calls (typically `list`) from growing the actions state and consuming RAM during long-running backups. Example: `[list]`
   cancel_operation_timeout: "1800s" # API_CANCEL_OPERATION_TIMEOUT, how long `/backup/kill` (and server stop/restart) waits for the underlying command goroutine to actually return after the context is canceled. If the goroutine is stuck on an IO without timeout, kill returns once this timeout elapses. See https://github.com/Altinity/clickhouse-backup/issues/1365