Kubernetes discovery: cap API calls at 30s and show retry error nodes (no built-in client timeout)

[tnaum-ms]

Summary

Kubernetes service discovery currently issues @kubernetes/client-node API calls without any request timeout. If the Kubernetes API server hangs the socket (VPN down, packet blackhole, wrong server URL behind a slow proxy), the discovery tree spins indefinitely (or until the OS TCP timeout, which can be minutes) with no way for the user to tell discovery from a hang.

This is the follow-up for finding 2.2 in the PR #621 pre-merge review (docs/ai-and-plans/PRs/621-kubernetes-discovery/pre-merge-code-review.md).

Decision (agreed)

• Cap every Kubernetes discovery API call at 30 seconds.
• On timeout, surface the existing error node with the "Click here to retry" action (same path already used for RBAC/network failures) — i.e. fail into the established retry/error-node UX, not a silent hang.
• Do NOT wire up AbortController/cancellation on node collapse. Collapsing the tree node will not abort in-flight requests; the only behavior we add is the 30s cap + retry error node.

Background: why this is needed

A research pass against the actual shipped @kubernetes/client-node v1.4.0 code in node_modules confirmed:

• No built-in default request timeout on the generated API clients (CoreV1Api, CustomObjectsApi, …). createConfiguration() has no timeout field; makeApiClient() passes none; the auth/agent hook sets none.
• The transport is node-fetch v2 (not undici), called with only method/body/headers/signal/agent — no timeout passed. So even undici's headersTimeout/bodyTimeout (~300s) defaults do not apply.
• Connect blackhole (SYN dropped, no RST) → hangs until the OS TCP connect timeout (~20s to >2min). Connected-but-silent server → hangs indefinitely.
• The generated API methods do not accept an AbortSignal argument. The timeoutSeconds? param on listNamespace(...) is the Kubernetes server-side list/watch query parameter — it does nothing on a connect blackhole.
• The supported way to inject a signal is a client-wide pre middleware on the API client built from KubeConfig that calls ctx.setSignal(AbortSignal.timeout(ms)). AbortSignal.timeout(ms) covers both the connect phase and the awaiting-response phase, so a single 30s signal bounds the whole operation.

Suggested implementation

• Attach a client-wide timeout middleware (AbortSignal.timeout(30_000)) where the API clients are constructed — createCoreApi() in src/plugins/service-kubernetes/kubernetesClient.ts, and the CustomObjectsApi client used for DKO dbs listing in the same file. This covers all discovery calls in one place:
  • coreApi.listNamespace() — kubernetesClient.ts
  • coreApi.listNamespacedService({ namespace }) — kubernetesClient.ts
  • customApi.listNamespacedCustomObject({ ... }) (DKO dbs) — kubernetesClient.ts
  • coreApi.readNamespacedEndpoints({ ... }) — portForwardTunnel.ts
  • secret reads (DKO + generic credential secrets) — kubernetesClient.ts
• The middleware approach is required because the generated methods don't take a per-call signal; a Promise.race wrapper alone would resolve our wrapper but leave the underlying socket hanging.
• On a timeout rejection, the existing getChildren() error handling in KubernetesContextItem.ts (and the namespace-level prescan catch) should produce the standard "Click here to retry" error node. A timeout error should be classified/logged distinctly in the [KubernetesDiscovery] output channel so it's debuggable.
• 30_000 ms as a constant (not a user setting), consistent with how NAMESPACE_PRESCAN_CONCURRENCY is a deliberate hardcoded non-setting. Revisit only if telemetry warrants.

Related: namespace pre-scan ceiling (review finding 2.5)

This work should also account for the per-namespace pre-scan in KubernetesContextItem.ts, which fans out DKO + service listing across namespaces with bounded concurrency (NAMESPACE_PRESCAN_CONCURRENCY = 5) but no per-call time ceiling. Today a single slow/hung namespace ties up one of the 5 workers indefinitely and can stall the visible result on a large cluster.

• The 30s per-call cap above largely covers this once it applies to listNamespacedService / listNamespacedCustomObject, since each prescan worker's call will now reject after 30s into the per-namespace error/retry node instead of hanging.
• Consider as part of this fix: confirm the capped-parallelism prescan degrades gracefully under the new timeout — i.e. one timed-out namespace produces a per-namespace retry node and frees its worker so the remaining namespaces still complete, rather than the whole context appearing to hang. Add/adjust a test for "one namespace times out, others still render."
• This folds review finding 2.5 into this issue rather than tracking it separately.

Scope notes

• Applies to the discovery-tree read calls and the connect/port-forward resolution path (which also calls createCoreApi), so both are covered by capping at the client-construction site.
• Out of scope: cancellation-on-collapse (explicitly excluded), per-request configurable timeouts, and any change to the server-side timeoutSeconds list/watch parameter.

Acceptance criteria

• Kubernetes discovery API calls reject after ~30s when the API server is unreachable/hanging, instead of spinning indefinitely.
• A timed-out call renders the existing error node with "Click here to retry" (first child), consistent with other discovery error states.
• A single timed-out namespace in the pre-scan frees its worker and produces a per-namespace retry node; the other namespaces still render (covers 2.5).
• Timeout is logged to the [KubernetesDiscovery] output channel with enough context to debug.
• No cancellation/abort is wired to node collapse.
• Unit tests cover the timeout → error-node path (including the "one namespace times out, others still render" case).

References

• PR feat(kubernetes): add multi-source service discovery #621 — feat(kubernetes): add multi-source service discovery
• Pre-merge review doc: docs/ai-and-plans/PRs/621-kubernetes-discovery/pre-merge-code-review.md (findings 2.2 and 2.5)