fix(K8sPipelineClient): tolerate pod/job deserialization errors from newer Kubernetes (fixes HTTP 400 on ingestionPipelines/status)#29791
Conversation
…newer Kubernetes
On clusters newer than the bundled kubernetes `client-java` models, listing
pods/jobs throws an `IllegalArgumentException` while deserializing the response,
because the pod status carries fields the models don't define, e.g.:
The field `allocatedResources` in the JSON string is not defined in the
`V1PodStatus` properties
That `IllegalArgumentException` is a `RuntimeException`, not an `ApiException`, so
it was uncaught and surfaced as HTTP 400 from
`GET /services/ingestionPipelines/status` — which blanks the service
Agents/Ingestion tab in the UI. Reproduced on EKS Kubernetes 1.36 with
OpenMetadata 1.13 (client-java 24.0.0). The latest client-java (26.0.0) still
trails recent Kubernetes, so bumping the dependency is not a durable fix.
Make the three pod/job read paths resilient to client-side deserialization
failures (the API call itself succeeded, so cluster access is fine):
- getServiceStatusInternal(): report healthy instead of failing the status endpoint
- getQueuedPipelineStatusInternal(): treat as no queued jobs
- the ingestion log-fetch path: return a graceful "logs unavailable" message
This keeps K8sPipelineClient's UI working across Kubernetes upgrades instead of
breaking whenever the cluster adds a status field ahead of client-java.
Signed-off-by: Michael Black <michael.black@pura.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
✅ PR checks passedThe linked issue has a description and all required Shipping project fields set. Thanks! |
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
…errors) Per review feedback (gitar-bot): catching the whole RuntimeException hierarchy could swallow genuine bugs (NPE, IllegalStateException, uninitialized API clients) and, in getServiceStatusInternal(), report an unhealthy cluster as healthy. Narrow all three handlers to IllegalArgumentException — the exact exception client-java throws on unknown pod/job status fields — so real runtime failures still propagate while version-skew deserialization degrades gracefully. Signed-off-by: Michael Black <michael.black@pura.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
| } catch (IllegalArgumentException e) { | ||
| // The list calls above reached the API server successfully (any real | ||
| // access/RBAC failure would be an ApiException, handled above). An | ||
| // IllegalArgumentException here is thrown by the bundled Kubernetes client | ||
| // (client-java) when it cannot DESERIALIZE a pod/job in the response — | ||
| // typically because the | ||
| // cluster runs a newer Kubernetes than the client models and the response | ||
| // carries an unmodeled status field, e.g.: | ||
| // IllegalArgumentException: The field `allocatedResources` in the JSON | ||
| // string is not defined in the `V1PodStatus` properties | ||
| // Cluster access is fine, so report healthy instead of failing the whole | ||
| // /services/ingestionPipelines/status endpoint (which blanks the UI). | ||
| LOG.warn( | ||
| "Kubernetes namespace {} is reachable but a pod/job could not be deserialized " | ||
| + "(client-java models likely lag the cluster's Kubernetes version): {}", | ||
| namespace, | ||
| e.getMessage()); | ||
| return buildHealthyStatus(getKubernetesVersion()) | ||
| .withReason(String.format(K8S_AVAILABLE_FORMAT, namespace, serviceAccount)); |
There was a problem hiding this comment.
| + "the cluster's Kubernetes version): {}", | ||
| pipelineName, | ||
| e.getMessage()); | ||
| return Map.of("logs", FAILED_LOGS_MESSAGE + e.getMessage()); |
There was a problem hiding this comment.
logs payload key. The log response path treats that key as stream content, so users receive a regular log line containing the error message and the stream ends normally. A Kubernetes client parsing failure should be returned in an error-shaped response, not mixed into pipeline output.
harshach
added
the
safe to test
| } catch (IllegalArgumentException e) { | ||
| // The list calls above reached the API server successfully (any real | ||
| // access/RBAC failure would be an ApiException, handled above). An | ||
| // IllegalArgumentException here is thrown by the bundled Kubernetes client | ||
| // (client-java) when it cannot DESERIALIZE a pod/job in the response — | ||
| // typically because the | ||
| // cluster runs a newer Kubernetes than the client models and the response | ||
| // carries an unmodeled status field, e.g.: | ||
| // IllegalArgumentException: The field `allocatedResources` in the JSON | ||
| // string is not defined in the `V1PodStatus` properties | ||
| // Cluster access is fine, so report healthy instead of failing the whole | ||
| // /services/ingestionPipelines/status endpoint (which blanks the UI). | ||
| LOG.warn( | ||
| "Kubernetes namespace {} is reachable but a pod/job could not be deserialized " | ||
| + "(client-java models likely lag the cluster's Kubernetes version): {}", | ||
| namespace, | ||
| e.getMessage()); | ||
| return buildHealthyStatus(getKubernetesVersion()) | ||
| .withReason(String.format(K8S_AVAILABLE_FORMAT, namespace, serviceAccount)); |
There was a problem hiding this comment.
| } catch (IllegalArgumentException e) { | ||
| // client-java could not deserialize a pod returned by a newer Kubernetes | ||
| // than its models (see getServiceStatusInternal). Return a graceful message | ||
| // instead of propagating the error. | ||
| LOG.warn( | ||
| "Could not deserialize pod while fetching logs for pipeline {} (client-java lags " | ||
| + "the cluster's Kubernetes version): {}", | ||
| pipelineName, | ||
| e.getMessage()); | ||
| return Map.of("logs", FAILED_LOGS_MESSAGE + e.getMessage()); |
There was a problem hiding this comment.
logs key. The log response path treats that key as stream content, so users receive the Kubernetes parsing error as an ordinary pipeline log line and the stream ends normally. A client-side parsing failure should use an error-shaped response instead of being mixed into pipeline output.
Code Review ✅ Approved 1 resolved / 1 findingsAdds graceful handling for deserialization errors in K8sPipelineClient, ensuring UI functionality on newer Kubernetes clusters. Resolved the broad exception handling by narrowing the catch clause to IllegalArgumentException. ✅ 1 resolved✅ Bug: Broad catch(RuntimeException) masks real errors; reports cluster healthy
OptionsDisplay: compact → Showing less information. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Gitar |
|
🟡 Playwright Results — all passed (20 flaky)✅ 4090 passed · ❌ 0 failed · 🟡 20 flaky · ⏭️ 22 skipped
🟡 20 flaky test(s) (passed on retry)
How to debug locally# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip # view trace |
2 participants
Fixes #29792
Problem
On Kubernetes clusters newer than the bundled
io.kubernetes:client-javamodels,K8sPipelineClientfails to read pod/job status. Listing pods/jobs throws anIllegalArgumentExceptionduring response deserialization because the pod status carries fields the client models don't define:That exception is a
RuntimeException, not anApiException, so it is uncaught and surfaces as HTTP 400 fromGET /api/v1/services/ingestionPipelines/status. In the UI this blanks the database service Agents/Ingestion tab (no pipelines/status render). It also affects the queued-status and ingestion-log paths, which parse pods the same way.Environment / repro
client-java24.0.0),pipelineServiceClientConfig.type: k8sGET /api/v1/services/ingestionPipelines/status→ 400Why not just bump
client-java?allocatedResources/observedGenerationare recent Kubernetes pod-status additions. The latestclient-java(26.0.0, ~k8s 1.34) still trails current clusters (e.g. 1.36), and the client's strict "reject unknown fields" deserialization means a version bump only defers the same break to the next Kubernetes release (a treadmill). A public lenient/ignore-unknown-fields switch is not exposed byclient-java(theisLenientOnJsonflag has no public setter).Fix
Make the three pod/job read paths in
K8sPipelineClientresilient to client-side deserialization failures. ARuntimeExceptionfrom these calls means the API server responded (real access/RBAC errors areApiException, still handled) but the bundled models couldn't parse a newer-Kubernetes field — cluster access is fine, so degrade gracefully instead of failing:getServiceStatusInternal()— report healthy instead of 400-ing the status endpointgetQueuedPipelineStatusInternal()— treat as no queued jobsThis keeps K8sPipelineClient's UI functional across Kubernetes upgrades rather than breaking whenever the cluster adds a status field ahead of
client-java.Notes
catch (RuntimeException e)clauses only; no behavior change on the success orApiExceptionpaths.openmetadata-serviceMaven build locally, so I'm relying on project CI. Happy to narrow the caught type (e.g.IllegalArgumentException) or add a unit test if maintainers prefer.🤖 Generated with Claude Code
Greptile Summary
This PR makes Kubernetes pipeline status and log reads tolerate newer cluster fields. The main changes are:
IllegalArgumentExceptionfrom queued job status reads.Confidence Score: 4/5
This is close, but the fallback behavior should be fixed before merging.
openmetadata-service/src/main/java/org/openmetadata/service/clients/pipeline/k8s/K8sPipelineClient.java
Important Files Changed
Reviews (3): Last reviewed commit: "Merge branch 'main' into fix/k8s-pipelin..." | Re-trigger Greptile
Context used (3)