Skip to content

[2.8] Warn on deploy prepare storage rewrites#4530

Open
YuanTingHsieh wants to merge 6 commits intoNVIDIA:2.8from
YuanTingHsieh:codex/deploy-prepare-storage-warnings
Open

[2.8] Warn on deploy prepare storage rewrites#4530
YuanTingHsieh wants to merge 6 commits intoNVIDIA:2.8from
YuanTingHsieh:codex/deploy-prepare-storage-warnings

Conversation

@YuanTingHsieh
Copy link
Copy Markdown
Collaborator

@YuanTingHsieh YuanTingHsieh commented May 6, 2026

What changed

  • Relocate Docker server job history and snapshot storage into the mounted workspace, matching the existing K8s server prepare behavior.
  • Share the server storage relocation helper across Docker and K8s prepare paths.
  • Warn when deploy prepare replaces/removes existing resource manager, resource consumer, or launcher configuration that would otherwise be silently discarded.
  • Warn when a present snapshot_persistor cannot be relocated because it does not expose snapshot_persistor.args.storage.args.root_dir; absent snapshot_persistor remains silent.

Why

Containerized deploy prepare rewrites runtime components and, for K8s, relocates server storage to durable workspace storage. Before this change, Docker server prepare left default job/snapshot stores under container-local /tmp/nvflare/..., and both component rewrites and malformed snapshot persistor layouts could silently drop operator intent.

Validation

  • python3 -m py_compile nvflare/tool/deploy/deploy_commands.py tests/unit_test/tool/deploy/deploy_commands_test.py

@YuanTingHsieh YuanTingHsieh changed the title [codex] Warn on deploy prepare storage rewrites [2.8] Warn on deploy prepare storage rewrites May 6, 2026
@YuanTingHsieh YuanTingHsieh marked this pull request as ready for review May 6, 2026 20:38
Copilot AI review requested due to automatic review settings May 6, 2026 20:38
@YuanTingHsieh YuanTingHsieh changed the base branch from main to 2.8 May 6, 2026 20:39
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves nvflare deploy prepare for containerized runtimes by relocating server-side job/snapshot storage into the mounted workspace (to align Docker with existing K8s behavior) and by surfacing warnings when prepare-time component rewrites would otherwise silently discard operator configuration.

Changes:

  • Relocate Docker server snapshot/job storage paths into the mounted workspace during deploy prepare (matching K8s behavior).
  • Share/extend the server storage relocation helper and add warnings for non-relocatable snapshot persistor shapes.
  • Warn when deploy prepare replaces/removes existing resource manager/consumer and launcher components/config during runtime preparation.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
nvflare/tool/deploy/deploy_commands.py Adds Docker server storage relocation, introduces warning helpers for discarded component config, and warns on unexpected snapshot persistor shapes.
tests/unit_test/tool/deploy/deploy_commands_test.py Adds unit tests for Docker server storage relocation and for warning/silence behavior around snapshot persistor shape and component replacement.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread nvflare/tool/deploy/deploy_commands.py
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented May 6, 2026

Greptile Summary

This PR adds two behavioral improvements to deploy prepare: (1) Docker server kits now relocate snapshot/job storage into the mounted workspace (matching existing K8s behavior), and (2) operators are warned when deploy prepare would silently discard existing resource manager, resource consumer, or launcher configuration. A warning is also emitted when a present snapshot_persistor cannot be relocated due to an unexpected nested key structure.

  • Storage relocation for Docker server: _relocate_server_storage_to_workspace is now called in _prepare_docker for server kits, sharing the helper already used in _prepare_k8s.
  • Component replacement warnings: _warn_for_replaced_components is called before the filtering in _patch_resources; it uses _component_has_custom_config and _launcher_replacement_discards_config to suppress noise on default/builtin configs.
  • Tests: New parametrized tests cover storage relocation, silent-no-snapshot-persistor, warn-on-bad-persistor-shape, warn-on-custom-component-replacement, and no-warn-on-builtin-defaults.

Confidence Score: 5/5

Safe to merge. The change adds operator-visible warnings and extends Docker server storage relocation; it does not alter any data paths or security boundaries.

The warning predicates correctly distinguish builtin defaults from custom configuration, with no false-positive risk on the default provisioned kit (which uses GPUResourceConsumer). Storage relocation for Docker server is a straightforward call to the same helper already exercised by the K8s path. The (KeyError, TypeError) guard is a strict improvement over the previous bare except KeyError. All new paths are covered by focused unit tests.

No files require special attention.

Important Files Changed

Filename Overview
nvflare/tool/deploy/deploy_commands.py Adds warning logic for replaced components, extends _relocate_server_storage_to_workspace error handling, and calls it from the Docker server path. Logic in _component_has_custom_config and _launcher_replacement_discards_config correctly distinguishes builtin defaults from custom configuration.
tests/unit_test/tool/deploy/deploy_commands_test.py Adds comprehensive tests for storage relocation, snapshot persistor warning/silence scenarios, component-replacement warnings, and no-warn-on-builtin-defaults cases. Test helpers and assertions cover both stdout and stderr correctly.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[prepare_deployment] --> B{runtime?}
    B -->|docker| C[_prepare_docker]
    B -->|k8s| D[_prepare_k8s]
    C --> E[_patch_resources]
    E --> W[_warn_for_replaced_components]
    W --> F[filter + rebuild components]
    F --> H{role == SERVER?}
    H -->|yes| I[_relocate_server_storage_to_workspace]
    H -->|no| J[continue]
    I --> K{snapshot_persistor present?}
    K -->|yes, good shape| L[set root_dir + uri_root]
    K -->|yes, bad shape| M[_warn: cannot relocate]
    K -->|absent| N[silent]
    D --> O[_patch_resources]
    O --> W2[_warn_for_replaced_components]
    W2 --> P[filter + rebuild components]
    P --> Q{role == SERVER?}
    Q -->|yes| I
    Q -->|no| R[continue]
Loading

Reviews (5): Last reviewed commit: "Merge branch '2.8' into codex/deploy-pre..." | Re-trigger Greptile

Comment thread nvflare/tool/deploy/deploy_commands.py Outdated
Comment thread nvflare/tool/deploy/deploy_commands.py Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants