feat: add KubeflowExecutor for Kubeflow Training Operator on Kubernetes

Introduces KubeflowExecutor and a matching TorchX scheduler so users can
deploy distributed training jobs to any Kubernetes cluster running the
Kubeflow Training Operator via run.run() / run.Experiment.

Supports both PyTorchJob (Training Operator v1) and TrainJob (Training
Operator v2, torch-distributed runtime) via the job_kind parameter.

Core features:
- KubeflowExecutor builds and submits PyTorchJob/TrainJob CRDs via the
  Kubernetes API (local kubeconfig with in-cluster fallback)
- TorchX scheduler persists job state and maps KubeflowJobState → AppState
- Workdir sync: package() copies job_dir to a PVC via a throw-away Alpine
  data-mover pod using kubectl cp (tar-based, no internet required)
- pull_results() syncs the PVC back to localhost after the job completes;
  auto-resolves job_dir from persisted scheduler state when the executor
  was not assign()ed directly (typical after run.run())
- Template-based launch.sh (kubeflow.sh.j2) hides torchrun details
- workdir_local_path merges local scripts into job_dir before PVC sync
- cancel(wait=True) polls until both CR and pods are fully terminated
- Data-mover pod inherits tolerations/affinity/imagePullSecrets so it
  can be scheduled alongside the training workload
- TrainJob podTemplateOverrides use container name "node" (matching the
  ClusterTrainingRuntime) so volumeMounts merge correctly

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>

Author: ko3n1g

.gitignore

@@ -183,3 +183,4 @@ _version.py
 
 # NeMo Run
 .nemo_run/
+local/

nemo_run/__init__.py

@@ -24,6 +24,7 @@
 from nemo_run.core.execution.base import Executor, ExecutorMacros, import_executor
 from nemo_run.core.execution.dgxcloud import DGXCloudExecutor
 from nemo_run.core.execution.docker import DockerExecutor
+from nemo_run.core.execution.kubeflow import KubeflowExecutor
 from nemo_run.core.execution.launcher import FaultTolerance, SlurmRay, SlurmTemplate, Torchrun
 from nemo_run.core.execution.lepton import LeptonExecutor
 from nemo_run.core.execution.local import LocalExecutor
@@ -66,6 +67,7 @@
     "Packager",
     "Partial",
     "Plugin",
+    "KubeflowExecutor",
     "run",
     "Script",
     "SkypilotExecutor",

nemo_run/core/execution/__init__.py

@@ -16,6 +16,7 @@
 from nemo_run.core.execution.dgxcloud import DGXCloudExecutor
 from nemo_run.core.execution.lepton import LeptonExecutor
 from nemo_run.core.execution.local import LocalExecutor
+from nemo_run.core.execution.kubeflow import KubeflowExecutor
 from nemo_run.core.execution.skypilot import SkypilotExecutor
 from nemo_run.core.execution.slurm import SlurmExecutor
 
@@ -25,4 +26,5 @@
     "SkypilotExecutor",
     "DGXCloudExecutor",
     "LeptonExecutor",
+    "KubeflowExecutor",
 ]