[Store] Fix: validate HA backend availability during config parsing#2111
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a validation mechanism for High Availability (HA) backends to ensure they are available based on the current build configuration. Key changes include the addition of the ValidateHABackendAvailability function, its integration into the HA backend specification building process, and updated flag descriptions. Additionally, a new unit test suite has been added to verify the availability logic for ETCD, Redis, and K8S backends. I have no feedback to provide.
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
|
|
||
| auto availability = ValidateHABackendAvailability(backend_type.value()); | ||
| if (availability != ErrorCode::OK) { | ||
| return tl::make_unexpected(availability); | ||
| } |
There was a problem hiding this comment.
Nice fix! Just noticed the same consistency check seems to be missing on the client side.
Client::ConnectToMaster goes through ParseHABackendSpec (client_service.cpp:192-210), which also parses the <backend>://... prefix and builds an HABackendSpec directly fed into ha::CreateLeaderCoordinator — a parallel parsing path to the BuildHABackendSpec you're fixing. Since the client binary is built independently from the master (with its own STORE_USE_REDIS / STORE_USE_ETCD flags), the same "configured redis:// but redis isn't compiled in" issue still hits the late-failure path on the client.
If it's not much trouble, would you mind adding the same ValidateHABackendAvailability call right after ParseHABackendType in ParseHABackendSpec? That way both entry points behave identically:
auto backend_type = ha::ParseHABackendType(...);
if (!backend_type.has_value()) {
return tl::make_unexpected(ErrorCode::INVALID_PARAMS);
}
auto availability = ha::ValidateHABackendAvailability(backend_type.value());
if (availability != ErrorCode::OK) {
return tl::make_unexpected(availability);
}
Happy to defer to a follow-up if you'd prefer to keep this PR tightly scoped — not a blocker.
There was a problem hiding this comment.
Added the same HA backend availability check on the client-side parsing path. ParseHABackendSpec() now calls ValidateHABackendAvailability() immediately after parsing the backend type and returns UNAVAILABLE_IN_CURRENT_MODE before coordinator creation when the backend is not compiled into the current binary. I also added test coverage for the client-side parsing behavior.
Head branch was pushed to by a user without write access
3913b6a to
c689f03
Compare
|
Thanks for adding the client-side check too, @Lin-z-w! Both parsing paths now behave consistently! The build-wheel-cu13 failures look unrelated to this change. This is good to merge from my side. |
Description
Validate the configured HA backend against the backends compiled into the current binary before constructing the HA backend spec.
Previously, configuring
ha_backend_type=redisin a build withoutSTORE_USE_REDISwas accepted during config parsing and failed later when creating the Redis leader coordinator. This change moves the availability check earlier intoBuildHABackendSpec(), returningUNAVAILABLE_IN_CURRENT_MODEduring HA backend config validation.Also updates the
--enable_haflag description to refer to the configured HA backend instead of etcd specifically.Module
mooncake-transfer-engine)mooncake-store)mooncake-ep)mooncake-integration)mooncake-p2p-store)mooncake-wheel)mooncake-pg)mooncake-rl)Type of Change
How Has This Been Tested?
cmake -S . -B buildcmake --build build --target ha_backend_availability_testclang-format-20 -style=file --dry-run -Werror mooncake-store/src/master.cpp mooncake-store/include/ha/ha_types.h mooncake-store/src/ha/leadership/master_service_supervisor.cpp mooncake-store/tests/ha/leadership/ha_backend_availability_test.cppctest --test-dir build -R ha_backend_availability_test --output-on-failureChecklist
./scripts/code_format.shbefore submitting.