Skip to content

fix: render TES task volumes as emptyDir in K8s executor jobs#1420

Closed
lbeckman314 wants to merge 1 commit into
developfrom
fix/k8s-tes-volumes
Closed

fix: render TES task volumes as emptyDir in K8s executor jobs#1420
lbeckman314 wants to merge 1 commit into
developfrom
fix/k8s-tes-volumes

Conversation

@lbeckman314
Copy link
Copy Markdown
Contributor

@lbeckman314 lbeckman314 commented May 19, 2026

Overview 🌀

This PR fixes TES task.Volumes support in the Kubernetes backend to comply with the TES spec: volumes must be initialized as empty directories shared between all executors in a task, not backed by the S3 PVC.

Current Behavior ❌

TES task.Volumes were mounted as subpath mounts on the S3-backed PVC (funnel-storage-<taskId>), the same volume used for inputs/outputs. This caused two bugs:

  1. If a volume path overlapped with the PVC mount point (e.g. /mnt/data), the PV contents were exposed instead of an empty directory.
  2. Volumes were not truly empty or shared between executors as independent scratch space — they were subpath slices of a persistent S3 bucket.

New Behavior ✔️

Each path in task.Volumes is now rendered as a Kubernetes emptyDir volume in the executor job pod spec. The emptyDir is mounted at the declared path in every executor container, so data written by one executor is readable by the next — matching Docker's -v behavior as required by the TES spec.

Additionally, the S3 PVC is only created when the task has inputs or outputs. A task with only volumes no longer provisions a PVC at all.

Changes

File Change
config/kubernetes/executor-job.yaml TES TaskVolumes render as emptyDir volumes with matching volumeMounts. PVC mounts are now only for inputs/outputs.
worker/kubernetes.go Added TaskVolumes []string field to KubernetesCommand; passed to template data.
worker/worker.go Filters mapper.Volumes to exclude TES task volume paths before passing to the K8s command (prevents double-mounting). NeedsPVC is now true only when inputs or outputs exist.
compute/kubernetes/backend.go PV/PVC creation condition updated to `inputs
compute/kubernetes/resources/job.go Same NeedsPVC fix for the worker job template.
worker/kubernetes_volumes_test.go Three new unit tests: emptyDir rendering, no-PVC for volumes-only tasks, PVC + emptyDir coexistence.

@lbeckman314 lbeckman314 self-assigned this May 19, 2026
@lbeckman314 lbeckman314 added the blocker blocks user interactions label May 19, 2026
@lbeckman314 lbeckman314 linked an issue May 19, 2026 that may be closed by this pull request
@nss10
Copy link
Copy Markdown
Contributor

nss10 commented May 21, 2026

For the sake of posterity, This PR doesn't address mounting of volumes that can be shared across multiple executor pods.

@lbeckman314
Copy link
Copy Markdown
Contributor Author

Closing this PR in favor of EFS-backed PVC's (can re-open in future if needed)

This PR was originally planned to add support for emptyDir TES Volumes

Now we're targeting EFS-backed TES volumes (that will still need PV/PVC allocation) to support POSIX operations currently being limited by S3-backed volumes...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

blocker blocks user interactions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Review/Revisit Funnel's S3 Working Directory Behavior

2 participants