Port selected 2.8 fixes to main

[YuanTingHsieh]

Summary

Port the selected 2.8 fixes back to main in 2.8 merge order:

• [2.8] Add warnings for missing study data mappings #4528 Add warnings for missing study data mappings
• [2.8] Update deploy prepare launcher docs #4538 Update deploy prepare launcher docs
• [2.8] Add clean_up parameter to recipe Run.get_result() #4550 Align Run.get_result() with the clean_up parameter spelling
• [2.8] Clarify remove_client token cleanup semantics #4561 Clarify remove_client token cleanup semantics
• [2.8] Respect visible GPUs in resource manager #4563 Respect CUDA_VISIBLE_DEVICES in the GPU resource manager
• [2.8] Fix Docker SJ workspace tmpfs permissions #4574 Fix Docker SJ workspace tmpfs permissions
• [2.8] Narrow client failure reporting #4576 Narrow client failure reporting for generic launcher execution errors
• [2.8] Fix tracking recipe integration test #4583 Fix tracking recipe integration test

[Copilot]

Pull request overview

Backports eight selected fixes from the 2.8 branch into main in merge order. These cover study-data warning diagnostics, deploy prepare documentation, recipe Run.get_result(clean_up=...) parameter alignment, hidden remove_client token-cleanup semantics, GPU resource manager respecting CUDA_VISIBLE_DEVICES, Docker SJ workspace tmpfs permission fix, narrowed client failure reporting, and a tracking recipe integration test fix.

Changes:

• Code fixes: GPU resource manager visibility filtering, Docker tmpfs permissions (0o555→0o1777), removal of JobReturnCode.EXECUTION_ERROR from authoritative client failure reporting, clean_up parameter rename, hidden remove_client command with clarified docstrings, and study-data warning logging.
• Docs/design updates around nvflare deploy prepare, Helm chart, launcher_spec/resource_spec separation, and removed Docker example references.
• Test updates and additions covering each of the above.

Reviewed changes

Copilot reviewed 49 out of 49 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
nvflare/recipe/run.py	Rename cleanup→clean_up; updated docstring.
nvflare/private/fed/server/training_cmds.py	Hide remove_client; clarify description.
nvflare/private/fed/server/server_engine.py, server_engine_internal_spec.py, client_manager.py	Docstring clarifications for token-only removal.
nvflare/private/fed/server/fed_server.py	Stop treating generic EXECUTION_ERROR as authoritative job failure.
nvflare/private/fed/client/client_executor.py	Drop EXECUTION_ERROR from REPORTABLE_JOB_FAILURES.
nvflare/fuel/flare_api/{flare_api,api_spec}.py	Clarify remove_client semantics.
nvflare/app_opt/job_launcher/{docker_launcher,k8s_launcher,study_data}.py	Tmpfs 1777; pass logger; warn on missing study mappings.
nvflare/app_common/resource_managers/gpu_resource_manager.py	Respect CUDA_VISIBLE_DEVICES integer indices for memory/count checks.
examples/docker/*	Update README, add docker.yaml, move job image config to launcher_spec with GPUs in resource_spec.
docs/**	Extensive doc/design updates aligning with the above.
tests/unit_test/** and tests/integration_test/**	Add/adjust test coverage for each fix.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[greptile-apps]

Greptile Summary

This PR ports seven targeted fixes from the 2.8 branch to main, covering GPU resource scheduling, Docker workspace permissions, study data warnings, token-cleanup semantics, and launcher error handling.

• GPU resource manager (gpu_resource_manager.py): CUDA_VISIBLE_DEVICES is now parsed at init time; the manager restricts itself to visible nvidia-smi indices and uses actual GPU IDs as resource keys, with correct per-GPU memory validation. An autouse fixture in the test file ensures CUDA_VISIBLE_DEVICES is always cleared between test cases.
• Docker tmpfs mode (docker_launcher.py): Root workspace tmpfs mode changed from 0o555 (read-only for non-root) to 0o1777 (sticky world-writable) so non-root job-container users can create ephemeral dirs like snapshot-storage.
• Study data warnings (study_data.py): load_study_data_file and resolve_study_dataset_mounts now accept an optional logger and emit WARNING-level messages when the data file is absent, empty, or missing a requested study entry — making silent mount omissions visible in logs.

Confidence Score: 5/5

Safe to merge; all changes are well-scoped bug fixes with matching unit tests and no regressions in the reviewed paths.

Every code change has direct test coverage. The GPU resource manager correctly maps visible device indices to host GPU IDs and validates memory only for selected GPUs. The tmpfs mode fix is straightforward. The narrowing of EXECUTION_ERROR reporting is consistent between client and server and tested with an explicit new test case. No untested paths or missing guards were found in the changed code.

No files require special attention.

Important Files Changed

Filename	Overview
nvflare/app_common/resource_managers/gpu_resource_manager.py	Added CUDA_VISIBLE_DEVICES support: resource IDs now reflect actual nvidia-smi GPU indices rather than 0..n-1, with correct memory mapping and boundary handling
nvflare/app_opt/job_launcher/docker_launcher.py	Changed tmpfs root mode from 0o555 to 0o1777 (sticky world-writable) so non-root job containers can create ephemeral dirs; passes logger to study data helpers
nvflare/app_opt/job_launcher/study_data.py	Added optional logger parameter to load_study_data_file and resolve_study_dataset_mounts; emits warnings when study file is missing, empty, or lacks a requested study entry
nvflare/private/fed/client/client_executor.py	Removed JobReturnCode.EXECUTION_ERROR from REPORTABLE_JOB_FAILURES so generic launcher errors are no longer escalated to the server as job failures
nvflare/private/fed/server/fed_server.py	Removed JobReturnCode.EXECUTION_ERROR from the server-side condition that triggers fail_run(), aligned with the client-side change
nvflare/recipe/run.py	Renamed get_result parameter from cleanup to clean_up to match exec_env.stop(); existing keyword-argument callers with cleanup= will receive TypeError
nvflare/fuel/flare_api/flare_api.py	Clarified remove_client docstring: releases active token only, does not stop the client or prevent reconnect
nvflare/private/fed/server/training_cmds.py	Hid remove_client admin command (visible=False) and updated its description to reflect token-release semantics
tests/integration_test/experiment_tracking_recipes_test.py	Fixed recipe integration test: replaced direct importlib model loading with class_path config dict and file-to-apps distribution, fixing the recipe job dispatch flow
tests/unit_test/app_common/resource_managers/gpu_resource_manager_test.py	Added extensive tests for CUDA_VISIBLE_DEVICES filtering: visible GPU count, memory checks by actual GPU ID, stop-at-invalid behavior, and empty-string handling

Sequence Diagram

sequenceDiagram
    participant Env as Environment
    participant GRM as GPUResourceManager.__init__
    participant Helper as _get_cuda_visible_device_indices
    participant NvSMI as nvidia-smi helpers

    GRM->>NvSMI: get_host_gpu_ids()
    NvSMI-->>GRM: host_gpu_ids (e.g. [0,1,2,3])
    GRM->>Helper: _get_cuda_visible_device_indices(host_gpu_ids)
    Helper->>Env: os.environ.get("CUDA_VISIBLE_DEVICES")
    alt Not set
        Env-->>Helper: None
        Helper-->>GRM: None → use all host_gpu_ids
    else Set to integers (e.g. "2,3")
        Env-->>Helper: "2,3"
        Helper-->>GRM: [2, 3]
    else Set to UUID / NoDevFiles
        Env-->>Helper: "GPU-abc..."
        Helper-->>GRM: None → fall back to all host_gpu_ids
    else Set to empty string
        Env-->>Helper: ""
        Helper-->>GRM: [] → zero GPUs available
    end
    GRM->>NvSMI: get_host_gpu_memory_total()
    NvSMI-->>GRM: memory list (indexed by host position)
    GRM->>GRM: build host_gpu_mem_by_id
    GRM->>GRM: "resource_gpu_ids = managed[:num_of_gpus]"
    GRM->>GRM: validate memory per selected GPU ID
    GRM->>GRM: "resources = {gpu_id: GPUResource(gpu_id, mem)}"

Loading

_{Reviews (3): Last reviewed commit: "Merge branch 'main' into codex/port-2-8-..." | Re-trigger Greptile}