prepare v2.8.0 in ChangeLog.md

Author: Slach

ChangeLog.md

@@ -1,12 +1,23 @@
 # v2.8.0
 NEW FEATURES
+- add `rebase` command and `POST /backup/rebase/{name}` API endpoint — copy `required` parts from the `required_backup` chain into a remote incremental backup via server-side `CopyObject` (with streaming fallback) and remove the `required_backup` dependency, so the incremental backup becomes a full one without re-uploading data from the ClickHouse host; per-table parallelism is controlled by `general.rebase_concurrency` (env `REBASE_CONCURRENCY`, default = `download_concurrency`); requires backups made with `upload_by_part: true` and the same `compression_format` across the chain, fix [#1344](https://github.com/Altinity/clickhouse-backup/issues/1344), [#1444](https://github.com/Altinity/clickhouse-backup/issues/1444)
+- add `general.rebase_before_remove_old_remote` (env `REBASE_BEFORE_REMOVE_OLD_REMOTE`, default `false`) — makes `backups_to_keep_remote` a strict limit: when deletion of old remote backups is blocked by `required_backup` links from kept backups, the oldest kept increment is rebased first (same as the `rebase` command) so the whole out-of-window chain becomes deletable; rebase failure is not fatal and falls back to the legacy keep-required behavior
+- add `--schedule` to `watch` command, `general.watch_schedules` config section (env `WATCH_SCHEDULES`, `;`-separated) and the `schedule` query argument to `POST /backup/watch` — named cron-driven backup chains in `name=<name>,full=<cron>[,increment=<cron>][,full_type=create|rebase][,delete_previous_cycle=true|false]` format (standard 5-field cron, optional leading seconds field, `@every`/`@daily` descriptors), can be specified multiple times, mutually exclusive with `--watch-interval`/`--full-interval`; `name` is added as a prefix to `watch_backup_name_template` to isolate backup chains, `full_type=rebase` creates the scheduled full backup as increment + `rebase` (server-side copy of the previous chain instead of a full re-upload), `delete_previous_cycle=true` deletes all older backups of the chain after a successful full backup, fix [#1354](https://github.com/Altinity/clickhouse-backup/issues/1354)
+- add `--list-parts` and `--partitions` (`--list-partitions`) flags to `tables` command — additionally list every physical part (name, `partition_id`, size) or the distinct partitions (`partition_id`, `partition`, parts count, size) for each table; against the live server the data comes from `system.parts`, against `--local-backup`/`--remote-backup` part names come from backup metadata (`partition_id` derived from the part name), fix [#959](https://github.com/Altinity/clickhouse-backup/issues/959), [#1333](https://github.com/Altinity/clickhouse-backup/issues/1333)
+- add `--rbac`, `--configs`, `--named-collections` flags to `server --watch` — the server command never passed them through, so watch mode backed up only data even though the standalone `watch` command and the REST API endpoints already supported them, fix [#955](https://github.com/Altinity/clickhouse-backup/issues/955)
 - add `rebalance` command and `POST /backup/rebalance/{name}` API endpoint (with `--tables` and `--dry-run` support) — move data parts inside a local backup between disks: hardlink from the live part when `system.parts` shows it on another disk; parts absent from `system.parts` are copied to the storage-policy disk of the same type with the most free space (`unreserved_space - keep_free_space` from `system.disks`), parts on a disk which is absent or left the policy always move; parts on object disks are skipped, fix [#1024](https://github.com/Altinity/clickhouse-backup/issues/1024)
 - add `gcs.allow_multipart_upload` (env `GCS_ALLOW_MULTIPART_UPLOAD`, default `false`, experimental) with `gcs.upload_concurrency` and `gcs.multipart_upload_min_size` (default 1GB) — files bigger than `multipart_upload_min_size` upload to GCS as multiple parallel parts (part size is `chunk_size`) via gRPC client and compose into the final object; not compatible with `endpoint`, `force_http`, `encryption_key`, fix [#1028](https://github.com/Altinity/clickhouse-backup/issues/1028)
 - add `gcs.allow_multipart_download` (env `GCS_ALLOW_MULTIPART_DOWNLOAD`, default `false`) with `gcs.download_concurrency` — download each file as parallel range reads (part size is `chunk_size`) into a temporary file, mirrors `s3.allow_multipart_download`, fix [#1028](https://github.com/Altinity/clickhouse-backup/issues/1028)
 - add `general.disable_environment_override` (settable ONLY in the config file, no environment variable name on purpose, default `false`) — when `true`, config values come only from the config file: all environment variables and the `--env` CLI flag are ignored during config loading; protects against accidental overrides such as Kubernetes service-discovery variables, fix [#1079](https://github.com/Altinity/clickhouse-backup/issues/1079)
 
+IMPROVEMENTS
+- write a gzip-compressed bbolt file manifest `manifest.bolt.gz` during `upload` and read it during `download`/`restore` to eliminate the per-part `Walk` (`ListObjectsV2` and analogs) calls to remote storage that dominated restore time on backups with many parts; memory stays bounded even with millions of files (batched writes on upload, mmap-backed prefix seek on read), with graceful fallback to `Walk` when the manifest is missing or unreadable, and manifest failures are non-fatal for the backup, fix [#1375](https://github.com/Altinity/clickhouse-backup/issues/1375), [#1380](https://github.com/Altinity/clickhouse-backup/issues/1380)
+- store per-part `size` in table metadata during `create` and check the required free disk space right after table metadata download, before downloading any data — the check respects `--tables`/`--partitions`, resolves sizes of `required` parts through the required-backup chain, and when `--hardlink-exists-files` is used doesn't count parts which can be hardlinked from existing local parts (matched by `hash_of_all_files` via `system.parts` or by CRC64 of `checksums.txt`), fix [#1268](https://github.com/Altinity/clickhouse-backup/issues/1268)
+- fail fast on canceled context in the retrier — SIGTERM / `POST /backup/kill` no longer trigger a useless extra retry attempt with a misleading `Will wait and retry` warning per goroutine; during API server shutdown log which in-flight command and `cancel_operation_timeout` the server is waiting for instead of blocking silently
+
 BUG FIXES
 - fix silent truncation of compressed backup uploads to S3 when `s3.check_sum_algorithm: CRC32` (Object Lock buckets): the multipart upload stopped at the declared size (sum of raw file sizes), dropping the tar headers/padding/trailer tail of the archive; the stream is now consumed until EOF, fix [#1471](https://github.com/Altinity/clickhouse-backup/issues/1471)
+- fix `rbac_conflict_resolution: ignore` during `restore --rbac` — RBAC objects from the backup that already exist on the server are now really ignored: their `access/*.sql` files are not copied over the live ones and the correlated replicated `uuid` znodes and name-to-uuid mappings are not restored into Keeper, fix [#1013](https://github.com/Altinity/clickhouse-backup/issues/1013)
 
 # v2.7.4
 

Manual.md

@@ -177,6 +177,21 @@ OPTIONS:
    --config value, -c value                   Config 'FILE' name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG]
    --environment-override value, --env value  override any environment variable via CLI parameter
    
+```
+### CLI command - rebalance
+```
+NAME:
+   clickhouse-backup rebalance - Move data parts inside local backup between disks to match current system.parts layout and storage policy, skip parts on object disks
+
+USAGE:
+   clickhouse-backup rebalance [-t, --tables=<db>.<table>] [--dry-run] <backup_name>
+
+OPTIONS:
+   --config value, -c value                   Config 'FILE' name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG]
+   --environment-override value, --env value  override any environment variable via CLI parameter
+   --table value, --tables value, -t value    Rebalance only database and objects which matched with table name patterns, separated by comma, allow ? and * as wildcard
+   --dry-run                                  Only log which parts would move between disks, change nothing
+   
 ```
 ### CLI command - restore
 ```