v2.7.1

Author: Slach

ChangeLog.md

@@ -1,7 +1,29 @@
 # v2.7.1
+
+NEW FEATURES
+- add `--fips-info` app-level flag — prints binary name, version, git commit, build date, Go version, and the FIPS module build/runtime state (`GOFIPS140` build setting, `GODEBUG fips140` default/runtime) then exits, without requiring a Go toolchain, fix [#1402](https://github.com/Altinity/clickhouse-backup/issues/1402)
+- add Azure AD Workload Identity support for `azblob` — when `AZBLOB_USE_MANAGED_IDENTITY=true` and the `AZURE_TENANT_ID`/`AZURE_CLIENT_ID`/`AZURE_FEDERATED_TOKEN_FILE` env vars are injected (e.g. by the AAD Workload Identity webhook), the federated token is used to authenticate; see `Examples.md` for deployment, fix [#1124](https://github.com/Altinity/clickhouse-backup/issues/1124)
+
+IMPROVEMENTS
 - add `general.compression_use_multi_thread` (env `COMPRESSION_USE_MULTI_THREAD`, default `false`), `general.compression_threads` (env `COMPRESSION_THREADS`, default `0` = auto/GOMAXPROCS) and `general.compression_buffer_size` (env `COMPRESSION_BUFFER_SIZE`, default `0`) config options to tune per-stream zstd/gzip threading and the compression buffer (zstd encoder window / gzip DEFLATE window / pgzip block size); per-stream compression is now single-threaded by default to avoid CPU over-subscription, since `upload_concurrency`/`download_concurrency` already parallelize across tables, fix [#1378](https://github.com/Altinity/clickhouse-backup/issues/1378)
 - migrate compression from the archived, frozen `github.com/mholt/archiver/v4 v4.0.0-alpha.8` to its maintained successor `github.com/mholt/archives v0.1.5`, removing the `replace` directive pinned in [archiver#428](https://github.com/mholt/archiver/issues/428)
+- migrate S3 storage from the deprecated `github.com/aws/aws-sdk-go-v2/feature/s3/manager` to `github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager`; the forced CRC32 `aws-chunked` trailer that broke non-AWS S3-compatible providers is now disabled via `RequestChecksumCalculation=WhenRequired`, and the obsolete `s3.buffer_size` / `S3_BUFFER_SIZE` option was removed, fix [#1409](https://github.com/Altinity/clickhouse-backup/pull/1409)
+- add buffer-size and HTTP-transport tuning for high-bandwidth (10Gbps+) S3/GCS transfers: `general.pipe_buffer_size` (env `PIPE_BUFFER_SIZE`, default 128KB), `general.download_copy_buffer_size` (env `DOWNLOAD_COPY_BUFFER_SIZE`), `gcs.upload_buffer_size` (env `GCS_UPLOAD_BUFFER_SIZE`), `s3.http_write_buffer_size` (env `S3_HTTP_WRITE_BUFFER_SIZE`) and `s3.http_read_buffer_size` (env `S3_HTTP_READ_BUFFER_SIZE`), fix [#1376](https://github.com/Altinity/clickhouse-backup/issues/1376)
+- replace post-hoc sleep-based bandwidth throttling with a token-bucket rate limiter wrapped around the storage `Reader`/`Writer` interfaces, so the configured limit is enforced continuously during transfer instead of after each chunk, fix [#934](https://github.com/Altinity/clickhouse-backup/issues/934), [#1377](https://github.com/Altinity/clickhouse-backup/issues/1377)
+- parallelize `ALTER TABLE ... UNFREEZE` after `backup create` instead of running it inline inside each table goroutine, so an UNFREEZE no longer holds an `upload_concurrency` slot and blocks the next table, fix [#1381](https://github.com/Altinity/clickhouse-backup/issues/1381)
+- add `clickhouse.parts_columns_batch_size` (env `CLICKHOUSE_PARTS_COLUMNS_BATCH_SIZE`, default `25`) to batch the `system.parts` lookups when computing `hash_of_all_files`, avoiding `Max query size exceeded` failures on tables with very many parts, fix [#1408](https://github.com/Altinity/clickhouse-backup/issues/1408)
+- resolve `required` data parts during `restore` the same way as during `download` — hardlink from the required backup on local disk when present, otherwise download the part from remote storage, fix [#1023](https://github.com/Altinity/clickhouse-backup/issues/1023)
+- `ResumeOperationsAfterRestart` now ignores `create.state2`/`restore.state2` files instead of failing the API server startup with `unknown command`; only `upload` and `download` are auto-resumed after a server restart, fix [#1083](https://github.com/Altinity/clickhouse-backup/issues/1083)
+- fail fast with a clear error instead of retrying for ~35s when a remote table metadata `.json` is missing during `download` (covers S3 `NoSuchKey`, GCS 404, Azure `BlobNotFound`, FTP/SFTP not-found) — a missing table file is a permanent broken-backup condition, not a transient one, fix [#1379](https://github.com/Altinity/clickhouse-backup/issues/1379)
+- emit a clear error on `--resume download` when the local backup exists but `download.state2` is missing (so it is unknown which parts are complete), instead of crashing or silently resuming on top of partial data, fix [#1383](https://github.com/Altinity/clickhouse-backup/issues/1383)
+- harden FIPS 140-3 verification: native Go `GOFIPS140=v1.0.0` checks, ACVP reproducibility tests, outbound S3 TLS rejection checks and container cleanup in CI/CD, fix [#1399](https://github.com/Altinity/clickhouse-backup/pull/1399), [#1401](https://github.com/Altinity/clickhouse-backup/pull/1401), [#1404](https://github.com/Altinity/clickhouse-backup/pull/1404)
+
+BUG FIXES
 - don't kill `clickhouse-backup server` with `Fatal`/`os.Exit` when the resumable state DB can't be written or read (e.g. `no space left on device`); the error now propagates so the server stays alive and returns it to the API client, while the CLI exits with a non-zero code, fix [#1172](https://github.com/Altinity/clickhouse-backup/issues/1172)
+- fix `backup create` failing with `part "<name>" not found in system.parts ... after FREEZE` when a ClickHouse cache disk (e.g. `s3_cache`) wraps an underlying S3 object disk — prefer the underlying disk name over the cache wrapper in `getDisksFromSystemDisks`, fix [#1396](https://github.com/Altinity/clickhouse-backup/issues/1396)
+- fix `--hardlink-exists-files` to also match parts whose `hash_of_all_files` is identical but that now live under a renamed table, fix [#1398](https://github.com/Altinity/clickhouse-backup/issues/1398)
+- fix `--table` combined with `--resume` on incremental backups: recursively downloading a required backup closed the parent `b.dst` connection and wiped the resumable state; the connection is now saved/restored, fix [#1384](https://github.com/Altinity/clickhouse-backup/issues/1384)
+- improve detection when `clickhouse-backup` runs on a host whose disks differ from `clickhouse-server`, instead of silently warning `doesn't contain tables for restore`, fix [#1037](https://github.com/Altinity/clickhouse-backup/issues/1037)
 
 # v2.7.0