Increase storage test timeouts for the Azure disk CSI driver#18502
Increase storage test timeouts for the Azure disk CSI driver#18502rifelpet wants to merge 1 commit into
Conversation
kubernetes-prow
Bot
added
the
size/S
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
/test pull-kops-azure-gossip-ha |
|
Would reducing the parallelism for Azure help more? |
Not sure. This doc suggests that too many concurrent operations can result in slow attach/detach times: but claims the issue is specific to the in-tree driver and suggests migrating to CSI, which we already use here. This is the only related upstream issue, and is also specific to in-tree: kubernetes-sigs/azuredisk-csi-driver#269 |
|
Unknown CLA label state. Rechecking for CLA labels. Send feedback to sig-contributor-experience at kubernetes/community. /check-cla |
kubernetes-prow
Bot
added
the
cncf-cla: yes
|
/test pull-kops-azure-gossip-ha |
1 similar comment
|
/test pull-kops-azure-gossip-ha |
The disk.csi.azure.com attach/detach operations are ARM control-plane calls that routinely take longer than the e2e framework's default TimeoutContext values. This causes PersistentVolume test flakes in the External Storage suite (pods stuck Pending waiting on volume mount, slow PV deletion) on the e2e-kops-azure-gossip-ha job. Override the relevant framework.TimeoutContext values via the external storage testdriver manifest's Timeouts map so the disk.csi.azure.com tests tolerate the slower Azure attach/provision/delete latencies.
e9275c0 to
92a0d4e
Compare
|
/test pull-kops-azure-gossip-ha |
|
@rifelpet: The following test failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/test pull-kops-azure-gossip-ha |
|
@rifelpet let's give kubernetes/test-infra#37384 and #18527 a try first. |
3 participants
What
Add a
Timeouts:block to the external-storage testdriver manifest fordisk.csi.azure.com(tests/e2e/csi-manifests/azure-disk/driver.yaml), overriding severalframework.TimeoutContextdefaults for the Azure disk storage e2e tests.Why
The
e2e-kops-azure-gossip-hajob flakes on PersistentVolume tests becausedisk.csi.azure.comattach/detach/provision are ARM control-plane operations that routinely exceed the e2e framework's default timeouts. Observed failures in theExternal Storage [Driver: disk.csi.azure.com]suite:subPath should support readOnly file specified in the volumeMountsubPath should support non-existent pathvolumes should allow exec of files on the volumeAll fail identically: the test pod is scheduled but stays
Pending/PodInitializingwaiting on the volume to attach+mount, then trips the defaultPodStarttimeout (Timed out after 300.001s). The root cause is Azure disk attach latency, not a product bug — confirmed against the failing runs:How
The Kubernetes external-storage test harness supports a per-driver
Timeoutsmap that overridesframework.TimeoutContext. This is the supported, configuration-only knob — no changes tok8s.io/kubernetes/test/e2ecode. kops already passes this manifest via--storage.testdriver(tests/e2e/pkg/tester/tester.go).Consuming code in kubernetes/kubernetes
These settings are read by the external storage test driver:
driverDefinition.Timeoutsfield — the manifest field these YAML keys map to.driverDefinition.GetTimeouts()— parses each value withtime.ParseDurationand overlays it onto the defaultframework.TimeoutContext.GetDriverTimeouts()/CustomTimeoutsTestDriver— how the storage suites obtain the (possibly overridden) timeouts.TimeoutContext/defaultTimeouts— the struct whose Go field names these keys must match, and the defaults being overridden.New values (defaults in parentheses):
Scope / not addressed
This only affects the
External Storage [Driver: disk.csi.azure.com]tests. The two[sig-apps] StatefulSetflakes on the same job (2067116858862997504, 2065546496547229696) share the same Azure-latency root cause but use hardcoded constants (StatefulSetTimeout = 10m) in upstreamtest/e2eand cannot be tuned via configuration; they would need a separate upstream change.Assisted by Opus 4.8