Skip to content

[Store] Fix: validate HA backend availability during config parsing#2111

Merged
ykwd merged 2 commits into
kvcache-ai:mainfrom
Lin-z-w:fix-ha-backend-availability-check
May 25, 2026
Merged

[Store] Fix: validate HA backend availability during config parsing#2111
ykwd merged 2 commits into
kvcache-ai:mainfrom
Lin-z-w:fix-ha-backend-availability-check

Conversation

@Lin-z-w

@Lin-z-w Lin-z-w commented May 15, 2026

Copy link
Copy Markdown
Collaborator

Description

Validate the configured HA backend against the backends compiled into the current binary before constructing the HA backend spec.

Previously, configuring ha_backend_type=redis in a build without STORE_USE_REDIS was accepted during config parsing and failed later when creating the Redis leader coordinator. This change moves the availability check earlier into BuildHABackendSpec(), returning UNAVAILABLE_IN_CURRENT_MODE during HA backend config validation.

Also updates the --enable_ha flag description to refer to the configured HA backend instead of etcd specifically.

Module

  • Transfer Engine (mooncake-transfer-engine)
  • Mooncake Store (mooncake-store)
  • Mooncake EP (mooncake-ep)
  • Integration (mooncake-integration)
  • P2P Store (mooncake-p2p-store)
  • Python Wheel (mooncake-wheel)
  • PyTorch Backend (mooncake-pg)
  • Mooncake RL (mooncake-rl)
  • CI/CD
  • Docs
  • Other

Type of Change

  • Bug fix
  • New feature
  • Refactor
  • Breaking change
  • Documentation update
  • Other

How Has This Been Tested?

  • cmake -S . -B build
  • cmake --build build --target ha_backend_availability_test
  • clang-format-20 -style=file --dry-run -Werror mooncake-store/src/master.cpp mooncake-store/include/ha/ha_types.h mooncake-store/src/ha/leadership/master_service_supervisor.cpp mooncake-store/tests/ha/leadership/ha_backend_availability_test.cpp
  • ctest --test-dir build -R ha_backend_availability_test --output-on-failure

Checklist

  • I have performed a self-review of my own code.
  • I have formatted my own code using ./scripts/code_format.sh before submitting.
  • I have updated the documentation.
  • I have added tests to prove my changes are effective.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a validation mechanism for High Availability (HA) backends to ensure they are available based on the current build configuration. Key changes include the addition of the ValidateHABackendAvailability function, its integration into the HA backend specification building process, and updated flag descriptions. Additionally, a new unit test suite has been added to verify the availability logic for ETCD, Redis, and K8S backends. I have no feedback to provide.

@codecov-commenter

codecov-commenter commented May 15, 2026

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 87.87879% with 4 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...re/src/ha/leadership/master_service_supervisor.cpp 0.00% 3 Missing ⚠️
mooncake-store/include/ha/ha_types.h 90.90% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

Comment on lines 41 to +45

auto availability = ValidateHABackendAvailability(backend_type.value());
if (availability != ErrorCode::OK) {
return tl::make_unexpected(availability);
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice fix! Just noticed the same consistency check seems to be missing on the client side.

Client::ConnectToMaster goes through ParseHABackendSpec (client_service.cpp:192-210), which also parses the <backend>://... prefix and builds an HABackendSpec directly fed into ha::CreateLeaderCoordinator — a parallel parsing path to the BuildHABackendSpec you're fixing. Since the client binary is built independently from the master (with its own STORE_USE_REDIS / STORE_USE_ETCD flags), the same "configured redis:// but redis isn't compiled in" issue still hits the late-failure path on the client.

If it's not much trouble, would you mind adding the same ValidateHABackendAvailability call right after ParseHABackendType in ParseHABackendSpec? That way both entry points behave identically:

auto backend_type = ha::ParseHABackendType(...);
if (!backend_type.has_value()) {
    return tl::make_unexpected(ErrorCode::INVALID_PARAMS);
}
auto availability = ha::ValidateHABackendAvailability(backend_type.value());
if (availability != ErrorCode::OK) {
    return tl::make_unexpected(availability);
}

Happy to defer to a follow-up if you'd prefer to keep this PR tightly scoped — not a blocker.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the same HA backend availability check on the client-side parsing path. ParseHABackendSpec() now calls ValidateHABackendAvailability() immediately after parsing the backend type and returns UNAVAILABLE_IN_CURRENT_MODE before coordinator creation when the backend is not compiled into the current binary. I also added test coverage for the client-side parsing behavior.

@stmatengss
stmatengss enabled auto-merge (squash) May 16, 2026 11:17
auto-merge was automatically disabled May 19, 2026 02:51

Head branch was pushed to by a user without write access

@Lin-z-w
Lin-z-w force-pushed the fix-ha-backend-availability-check branch from 3913b6a to c689f03 Compare May 19, 2026 03:05
@00fish0

00fish0 commented May 23, 2026

Copy link
Copy Markdown
Collaborator

Thanks for adding the client-side check too, @Lin-z-w! Both parsing paths now behave consistently!

The build-wheel-cu13 failures look unrelated to this change. This is good to merge from my side.

@ykwd
ykwd merged commit 4a17b1d into kvcache-ai:main May 25, 2026
16 of 19 checks passed
@Lin-z-w
Lin-z-w deleted the fix-ha-backend-availability-check branch May 25, 2026 09:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants