Skip to content

MANTA-5484 Allow manta-rebalancer to rebalance manta-buckets-api objects #25

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
cneira wants to merge 190 commits into
base: manta-rebalancer
Choose a base branch
from
Draft

MANTA-5484 Allow manta-rebalancer to rebalance manta-buckets-api objects #25

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
190 commits
Select commit Hold shift + click to select a range
8c44072
Add manta-buckets-mdapi client to rust-libmanta
cneira Dec 29, 2025
ce8dee7
Implement list_buckets Fast RPC in rust-libmanta mdapi client
cneira Dec 30, 2025
47758f6
Fix FromSql implementation to use bytes instead of PgValue
cneira Dec 31, 2025
9e4673f
CHG-022: Add mdapi discovery support to rust-sharkspotter
cneira Dec 31, 2025
12c1dba
Integrate buckets-mdapi client into manta-rebalancer
cneira Dec 30, 2025
8e05b4b
Fix mdapi_client compilation: add md5 dependency
cneira Dec 30, 2025
df3b98d
Fix md5 usage for version 0.7.0 API
cneira Dec 30, 2025
f28da42
Fix private field access in mdapi_client unit tests
cneira Dec 30, 2025
48ed76b
Add mdapitests target to Makefile
cneira Dec 30, 2025
8a560cf
Integrate mdapi client into job execution
cneira Dec 30, 2025
c213ba0
Fix compilation errors in mdapi job execution integration
cneira Dec 30, 2025
22f6a0f
Fix batch callback error type conversion
cneira Dec 30, 2025
50400b1
Add mdapi bucket auto-discovery infrastructure
cneira Dec 30, 2025
1a21c34
Update list_buckets call for new rust-libmanta API
cneira Dec 30, 2025
ac39c79
Fix mdapi_client tests to include single_bucket_mode field
cneira Dec 31, 2025
329e93f
CHG-021: Add hybrid metadata backend support for complete shark evacu…
cneira Dec 31, 2025
0caa91a
CHG-021: Complete mdapi integration for evacuate operations
cneira Dec 31, 2025
369a207
CHG-023: Add mdapi deployment configuration
cneira Dec 31, 2025
0c0564b
Add SAPI deployment configuration documentation for mdapi
cneira Jan 8, 2026
0f42697
Add InvalidState variant to InternalErrorCode
cneira Jan 9, 2026
d780013
Fix test compilation errors
cneira Jan 12, 2026
7e79382
Add multipart upload evacuation support to rebalancer
cneira Jan 23, 2026
b71a345
Fix rebalancer-legacy build compatibility with Rust 1.90
cneira Jan 26, 2026
f355f5f
Add unit tests for manta-buckets-api object evacuation
cneira Jan 26, 2026
5049844
Add unit tests for manta-buckets-api object handling in libmanta
cneira Jan 26, 2026
2f2e6d5
Implement mdapi RPC methods and add sharkspotter CLI args
cneira Jan 26, 2026
19e0bef
Fix mdapi discovery shark tracking and error handling
cneira Jan 26, 2026
1103d5d
Fix MPU part key test to match regex expectations
cneira Jan 26, 2026
1054c31
Add connection pooling to mdapi client
cneira Jan 26, 2026
d6685e0
Fix agent tests by using user-writable temp directory
cneira Jan 27, 2026
d63ba97
Fix manager config tests race condition with unique temp files
cneira Jan 28, 2026
bf91b92
Add mdapi connection pool tests and mark PostgreSQL-dependent tests
cneira Jan 28, 2026
c8e7e6b
Fix manager test failures from global logger and shared state conflicts
cneira Jan 28, 2026
033c58c
Mark manager binary tests as ignored (require PostgreSQL)
cneira Jan 28, 2026
ac5e266
Fix moray batch_unsupported_test port conflict race condition
cneira Jan 28, 2026
95cc62d
Mark sharkspotter integration tests requiring infrastructure as ignored
cneira Jan 28, 2026
bce6624
Fix libmanta mdapi doc examples to use ignore instead of no_run
cneira Jan 28, 2026
01cce1c
CHG-051 TASK-001: Convert config update test to channel-based mock
cneira Jan 28, 2026
b0508e4
CHG-051 TASK-002: Extract status aggregation and add mock tests
cneira Jan 28, 2026
df92499
CHG-051 TASK-003: Convert sharkspotter missing_all_args to clap test
cneira Jan 28, 2026
ca0c6e5
Add unit tests for rebalancer-legacy, sharkspotter, and update tarpau…
cneira Jan 29, 2026
9206d80
Add critical business logic tests for evacuate job and config
cneira Jan 29, 2026
8fb0a22
Fix integration test failures for MetadataBackend error classification
cneira Jan 29, 2026
528dea8
Fix content update discard and hybrid batch atomicity gap
cneira Jan 30, 2026
e1bd6ca
Fix code review findings: properties mismatch, etag guard, and filter…
cneira Jan 30, 2026
fd0c34a
Replace panic/exit with graceful error handling in rebalancer product…
cneira Jan 30, 2026
16789d4
Harden rebalancer against data corruption, security, and cascading fa…
cneira Jan 30, 2026
10fd09f
Align ObjectUpdate struct with mdapi server UpdateObjectPayload
cneira Jan 30, 2026
f0cba06
Fix MPU shark tracking: selective replacement instead of bulk overwrite
cneira Jan 30, 2026
100f512
Harden config/storinfo against panic, crash, and concurrency issues
cneira Jan 30, 2026
3de2bb2
Add DB transaction boundaries and warn on malformed object headers
cneira Jan 30, 2026
6084bf4
Extract shared header and shark conversion helpers in mdapi_client
cneira Jan 30, 2026
70e58a8
Type MPU upload record schema for compile-time field safety
cneira Jan 30, 2026
0170b0c
Reduce allocations and consolidate regex compilation
cneira Jan 30, 2026
df28769
Fix compiler warnings and cleanup unused code
cneira Feb 2, 2026
8439c9c
Fix panic! macro deprecation warnings in agent_test_util
cneira Feb 2, 2026
47a8a4b
Fix derive helper attribute order in jobs/mod.rs
cneira Feb 2, 2026
375c0f1
Replace intersperse with join in ZkConnectString::to_string
cneira Feb 2, 2026
ee98714
Replace deprecated Error::description() with Display trait
cneira Feb 2, 2026
7fd9e74
Replace deprecated chrono::from_timestamp with from_timestamp_opt
cneira Feb 2, 2026
53b97cd
Fix listBuckets RPC method name to lowercase
cneira Feb 2, 2026
c5c2039
Fix all camelCase method names in mdapi client to lowercase
cneira Feb 2, 2026
38376ca
Make shark list retry delay configurable
cneira Feb 2, 2026
e9b4e17
Add table existence check before evacuation operations
cneira Feb 2, 2026
11329e5
Clarify batched inserts status (MANTA-4464 already implemented)
cneira Feb 2, 2026
88c5f44
Handle partial failure in mdapi batch updates safely
cneira Feb 2, 2026
8e68b06
Add mdapi_vnodes config option for bucket discovery
cneira Feb 2, 2026
76f575b
Add batch size limits and timeout configuration for mdapi
cneira Feb 2, 2026
15628a7
Add integration test framework for mdapi evacuation
cneira Feb 2, 2026
b9e4a3a
Add retry configuration with exponential backoff for mdapi
cneira Feb 2, 2026
fd1680b
Fix slog-scope logger propagation in spawned threads
cneira Feb 3, 2026
d3059d1
Fix flaky available_mb test race condition
cneira Feb 3, 2026
a631ecd
Add dual workspace configuration for legacy and modern crates
cneira Feb 5, 2026
d07afeb
Add Makefile targets for legacy workspace builds
cneira Feb 5, 2026
5df5a85
Preserve Cargo.lock during legacy builds and regenerate for modern
cneira Feb 5, 2026
1ede44c
Add coverage-legacy Makefile target
cneira Feb 5, 2026
32ec80b
Fix slog-scope logger initialization in mdapi_client tests
cneira Feb 6, 2026
6ef9305
Fix slog-scope logger initialization in tests for parallel execution
cneira Feb 6, 2026
15383e0
rebalancer: add mdapi owners and vnodes to SAPI template
cneira Feb 10, 2026
7a489e6
Rebalancer mdapi_client uses native batchupdateobjects RPC
cneira Feb 13, 2026
b585ee5
Wire listvnodes/listowners discovery RPCs into evacuation
cneira Feb 23, 2026
9495ceb
Wire mdapi_endpoint into sharkspotter and fix legacy build
cneira Feb 23, 2026
6527d14
Add DNS SRV discovery for mdapi, remove hardcoded port 2030
cneira Feb 26, 2026
ee0a366
rebalancer: derive mdapi shards from BUCKETS_MORAY_SHARDS
cneira Feb 27, 2026
b56090c
Add mock Fast RPC mdapi server for integration tests
cneira Feb 27, 2026
fa95212
Add rsync-to dev tool for quick rebalancer deployment
cneira Feb 27, 2026
62dc6f7
Set LD_LIBRARY_PATH for PostgreSQL in rebalancer SMF manifest
cneira Feb 27, 2026
e1504cc
Isolate rebalancer-agent build with separate target dir
cneira Feb 27, 2026
753632b
build-legacy: exclude agent from workspace build, build it separately
cneira Feb 27, 2026
7bcea53
rsync-to: fix agent binary path to use TOP not WORKSPACE_ROOT
cneira Feb 27, 2026
c7679d7
Build agent as standalone workspace to prevent feature unification
cneira Feb 27, 2026
483fe97
rsync-to: remove fallback to workspace target for agent binary
cneira Feb 27, 2026
8341ee3
sharkspotter: include hostname in error messages from directdb
cneira Feb 27, 2026
45c5ce5
rebalancer: make direct_db configurable, default to false
cneira Feb 27, 2026
04764ea
rebalancer: add MDAPI v2 storage path support
cneira Mar 2, 2026
8f44a14
rebalancer agent: improve AgentFSError logging with paths
cneira Mar 2, 2026
fc70038
rebalancer: fix mdapi list RPC deserialization and cross-device rename
cneira Mar 3, 2026
98a1bba
rebalancer: fix mdapi evacuation pipeline end-to-end
cneira Mar 3, 2026
f67fdda
rebalancer: fix mdapi transient error retry for metadata updates
cneira Mar 4, 2026
de8846c
pgclone: modernize as single-file multi-command tool
cneira Mar 4, 2026
ff77207
sharkspotter: add direct DB discovery for buckets-postgres
cneira Mar 4, 2026
6b94d30
rebalancer: wire direct_db_buckets config and fix SAPI template defaults
cneira Mar 4, 2026
af8349e
direct-db flag
cneira Mar 18, 2026
aff09ff
Revert cueball Rust 1.90 compat changes: not needed for SmartOS build
cneira Mar 19, 2026
2f6eb4f
sharkspotter: restore original comments from master, simplify directd…
cneira Mar 19, 2026
0d8cfa7
sharkspotter: unify direct_db flag, extract run_mdapi_shard_thread, a…
cneira Mar 19, 2026
5605987
rebalancer: remove direct_db_buckets, buckets_min/max_shard from mana…
cneira Mar 19, 2026
092b7f3
sharkspotter: fix directdb_buckets uuid column type mismatch
cneira Mar 19, 2026
9dd6052
sharkspotter: add summary logging for directdb discovery, pass raw ho…
cneira Mar 19, 2026
45474e0
pgclone: fix buckets alias regex double-backslash escaping
cneira Mar 19, 2026
97c0910
rebalancer: parallelize assignment checker polling (8 concurrent)
cneira Mar 19, 2026
7514c8a
rebalancer: add alternate source fallback for object downloads
cneira Mar 19, 2026
6a52b6d
rebalancer: drain remaining PostProcessing objects on broker exit
cneira Mar 20, 2026
7112678
rebalancer: add debug logging to mdapi put_object
cneira Mar 20, 2026
4c187d8
rebalancer: fix metadata update worker premature exit on empty queue
cneira Mar 20, 2026
70dec8f
rebalancer: fix DyanmicWorkerMsg typo to DynamicWorkerMsg
cneira Mar 20, 2026
31ce179
rebalancer: fix non-batched metadata updates not marking objects Comp…
cneira Mar 20, 2026
43767c8
rebalancer: spawn multiple drain workers for remaining assignments
cneira Mar 20, 2026
457e7b1
rebalancer: use recv_timeout in metadata broker to respawn idle workers
cneira Mar 20, 2026
1f56b30
rebalancer: add error_breakdown and skip_breakdown to job status API
cneira Mar 20, 2026
a860538
rebalancer: distinguish moray vs mdapi metadata update failures
cneira Mar 20, 2026
c28b451
pgclone: accept buckets-mdapi in domain validation and replace
cneira Mar 20, 2026
81a8c24
pgclone: fix buckets clone alias/registrar and VMAPI destroy
cneira Mar 20, 2026
91348d5
rebalancer: remove alternate_sources fallback
cneira Mar 20, 2026
e863d73
rebalancer: document agent concurrency tunables in README
cneira Mar 20, 2026
33bcdfb
rebalancer: add testing guide with pgclone setup, SQL queries, and er…
cneira Mar 20, 2026
3473b36
MANTA-4519 rebalancer manager thread could choose sharks that do not …
cneira Mar 23, 2026
cef87e3
rebalancer: add overview doc, fix direct_db default to true
cneira Mar 25, 2026
22686e6
add rebalancer overview, review comments
cneira Mar 30, 2026
45fbf58
devops operations guide
cneira Mar 30, 2026
3d83168
rebalancer: fix assignment checker infinite polling and pgclone alias…
cneira Mar 30, 2026
7a191f7
rebalancer: add agent backpressure to prevent assignment queue overflow
cneira Mar 31, 2026
fb63ca3
rebalancer-agent: bump default workers from 1 to 4
cneira Mar 31, 2026
ba34e5d
rebalancer: retry with backoff on agent 503 instead of skipping
cneira Mar 31, 2026
031234b
rebalancer-agent: increase max_queued from workers*3 to workers*100
cneira Mar 31, 2026
52f2622
rebalancer-agent: set max_queued to unlimited, rely on checker timeout
cneira Mar 31, 2026
0a434a3
rebalancer: bounded queue (workers*50) + increase max_assignment_age …
cneira Apr 1, 2026
f4fe38a
docs: document no in-job retries and multi-run evacuation strategy
cneira Apr 1, 2026
82ef7fe
rebalancer: mark stale Running/Setup jobs as Failed on startup
cneira Apr 1, 2026
167b44a
docs: document job failure on rebalancer restart and recovery procedure
cneira Apr 1, 2026
422d30c
rebalancer: add SourceObjectNotFound skip reason for 404s
cneira Apr 1, 2026
af12b35
docs: clarify etag errors — expected on re-runs, fresh pgclones fix them
cneira Apr 1, 2026
d437e36
rebalancer-agent: retry transient download errors before skipping
cneira Apr 3, 2026
d868bb3
docs: update ops guide with all tuning changes from this session
cneira Apr 3, 2026
15e11fd
pgclone: support multiple shards with repeated --moray-vm/--buckets-vm
cneira Apr 6, 2026
1108f8e
pgclone: add discover command to find postgres VMs across all CNs
cneira Apr 6, 2026
2366558
docs: remove stale buckets_min/max_shard from config reference
cneira Apr 6, 2026
3a217d2
docs: document multi-shard deployments across all guides
cneira Apr 6, 2026
fd6b15e
docs: remove all references to direct_db_buckets
cneira Apr 6, 2026
d115b66
docs: fix stale defaults, typos, and incomplete skip reason reference
cneira Apr 6, 2026
c0e575d
docs: fix 503 backoff description and rebalancer-adm command syntax
cneira Apr 6, 2026
a994689
docs: add verification, orphan cleanup, and capacity planning sections
cneira Apr 6, 2026
77dceeb
docs: add developer architecture guide with code references
cneira Apr 6, 2026
d7689c5
docs: remove line number references from developer architecture guide
cneira Apr 6, 2026
2144a1d
docs: fix threading diagram — connect sharkspotter translator
cneira Apr 6, 2026
7fca477
docs: redraw threading diagram with clear connections
cneira Apr 6, 2026
1a7ec72
docs: rewrite threading model as step-by-step pipeline
cneira Apr 6, 2026
6332b8c
docs: add step numbers to threading diagram, explain ordering below
cneira Apr 6, 2026
a08be5c
docs: merge unique content from operators_guide, delete it
cneira Apr 6, 2026
ebba038
docs: restore original operators_guide content into devops guide
cneira Apr 6, 2026
96f551b
docs: add shard distribution guide; fix rsync-to for multi-CN deploym…
cneira Apr 6, 2026
62bae74
remove scripts/ from tracking
cneira Apr 6, 2026
b32ead4
fix: rsync-to vmadm quoting and hardcoded VMAPI domain
cneira Apr 6, 2026
653b92f
fix: use bash arrays for SSH_OPTS in rsync-to
cneira Apr 6, 2026
66c2e73
fix: restore original rsync-to fast path, add remote CN as fallback
cneira Apr 6, 2026
4207b36
update rebalancer config for multi-shard dc1, add mpu-parts TODO, fixes
cneira Apr 6, 2026
df9d9d1
add configurable download timeout, fix rsync-to multi-CN loop
cneira Apr 7, 2026
acc360c
docs: update mpu-parts TODO, add status SQL to testing guide
cneira Apr 7, 2026
c1b3499
classify orphaned .mpu-parts skips as mpu_part_no_data
cneira Apr 7, 2026
96b5f63
fix test failures and remove hardcoded dev config from git
cneira Apr 10, 2026
c040ee4
remove --test-threads=1, fix parallel test execution
cneira Apr 10, 2026
5df2ee7
add images/rebalancer for zone image builds from monorepo
nshalman Apr 14, 2026
94db772
fix build: LD_LIBRARY_PATH for PG, blacklist→blocklist rename
nshalman Apr 14, 2026
8df729c
add Jenkinsfile for rebalancer image builds
nshalman Apr 14, 2026
f8222b2
gitignore: add image build artifacts
nshalman Apr 14, 2026
a8c5228
restore the rest of the Jenkinsfile
nshalman Apr 15, 2026
b366d31
update eng submodule: buildimage fixes + monorepo support
nshalman Apr 15, 2026
f872f1b
XXX KEBE SAYS LET'S TRY JENKINS
danmcd Apr 15, 2026
5b9c914
update to 24.4.1 agents
nshalman Apr 15, 2026
5cb441d
Try to tell rustup where our certs are
nshalman Apr 15, 2026
4103041
For now just run the manta-rebalancer image build out of the root Jen…
nshalman Apr 15, 2026
7a08073
Fix missing newline in Makefile
nshalman Apr 15, 2026
238e364
Test installing flex
nshalman Apr 15, 2026
42ac02d
Switch to newer base image
nshalman Apr 15, 2026
9381590
explicitly choose the BUILD_PLATFORM
nshalman Apr 15, 2026
b43e6aa
add openssl, sqlite3 and readline packages to zone image
cneira Apr 16, 2026
4d73d0a
static link for openssl, storage nodes have old version of openssl
cneira Apr 16, 2026
9f668a5
skip system-managed paths during evacuation to eliminate etag errors
cneira Apr 17, 2026
7f14508
support "none" sentinel to disable exclude_key_prefixes filter
cneira Apr 17, 2026
b182f74
fix SAPI template for exclude_key_prefixes and add decommission valid…
cneira Apr 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,16 @@
/target
/libs/rebalancer-legacy/target-agent
/cache
/rust
.env
.idea/
libs/rebalancer-legacy/manager/src/config.json

# Image build artifacts (eng buildimage/release targets)
images/*/bits/
images/*/cache/
images/*/proto/
images/*/make_stamps/
/make_stamps/
images/*/*.tar.gz
images/*/*.manifest
6 changes: 6 additions & 0 deletions .gitmodules
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
[submodule "deps/eng"]
path = deps/eng
url = https://github.com/TritonDataCenter/eng
[submodule "deps/manta-scripts"]
path = deps/manta-scripts
url = https://github.com/TritonDataCenter/manta-scripts
[submodule "deps/postgresql12"]
path = deps/postgresql12
url = https://github.com/postgres/postgres.git
Loading