fix: render TES task volumes as emptyDir in K8s executor jobs#1420
Closed
lbeckman314 wants to merge 1 commit into
Closed
fix: render TES task volumes as emptyDir in K8s executor jobs#1420lbeckman314 wants to merge 1 commit into
lbeckman314 wants to merge 1 commit into
Conversation
Contributor
|
For the sake of posterity, This PR doesn't address mounting of volumes that can be shared across multiple executor pods. |
Contributor
Author
|
Closing this PR in favor of EFS-backed PVC's (can re-open in future if needed) This PR was originally planned to add support for Now we're targeting EFS-backed TES volumes (that will still need PV/PVC allocation) to support POSIX operations currently being limited by S3-backed volumes... |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview 🌀
This PR fixes TES
task.Volumessupport in the Kubernetes backend to comply with the TES spec: volumes must be initialized as empty directories shared between all executors in a task, not backed by the S3 PVC.Current Behavior ❌
TES
task.Volumeswere mounted as subpath mounts on the S3-backed PVC (funnel-storage-<taskId>), the same volume used for inputs/outputs. This caused two bugs:/mnt/data), the PV contents were exposed instead of an empty directory.New Behavior ✔️
Each path in
task.Volumesis now rendered as a KubernetesemptyDirvolume in the executor job pod spec. TheemptyDiris mounted at the declared path in every executor container, so data written by one executor is readable by the next — matching Docker's-vbehavior as required by the TES spec.Additionally, the S3 PVC is only created when the task has
inputsoroutputs. A task with onlyvolumesno longer provisions a PVC at all.Changes
config/kubernetes/executor-job.yamlTaskVolumesrender asemptyDirvolumes with matchingvolumeMounts. PVC mounts are now only for inputs/outputs.worker/kubernetes.goTaskVolumes []stringfield toKubernetesCommand; passed to template data.worker/worker.gomapper.Volumesto exclude TES task volume paths before passing to the K8s command (prevents double-mounting).NeedsPVCis nowtrueonly when inputs or outputs exist.compute/kubernetes/backend.gocompute/kubernetes/resources/job.goNeedsPVCfix for the worker job template.worker/kubernetes_volumes_test.go