Fixes #25666: Add retry-with-backoff to getServiceStatus for transient failure recovery by RajdeepKushwaha5 · Pull Request #27112 · open-metadata/OpenMetadata

RajdeepKushwaha5 · 2026-04-07T02:51:11Z

Describe your changes:

Problem: PipelineServiceClient.getServiceStatus() delegates to getServiceStatusInternal() with zero retry logic. A single transient REST failure (e.g., selector manager closed during an Airflow restart or network blip) immediately returns an unhealthy response (code 500), causing OpenMetadata to mark the Airflow agent as UNAVAILABLE. The agent never auto-recovers — a manual UI refresh or service restart is required.

Root Cause: While a getServiceStatusBackoff() method with Resilience4j retry already existed in PipelineServiceClient, it was never called by any production code. Both callers (IngestionPipelineResource.getRESTStatus() and SystemRepository.getPipelineServiceClientValidation()) call getServiceStatus() directly, which had no retry.

Fix: Move the Resilience4j retry-with-backoff logic into getServiceStatus() itself, so every health check benefits from transient failure tolerance:

getServiceStatus() now retries up to 3 attempts with 5-second backoff intervals when the response code is not 200
getServiceStatusBackoff() is simplified to delegate to getServiceStatus() (avoids nested/double retry)
Removed the unused Supplier import

Test: Added getServiceStatusRecoversFromTransientFailure test that enqueues a transient 500 followed by a healthy 200 and verifies the final status is healthy with exactly 3 requests (detection + failed attempt + successful retry).

Type of change:

Checklist:

I have read the CONTRIBUTING document.
My PR title is Fixes <issue-number>: <short explanation>
I have commented on my code, particularly in hard-to-understand areas.
For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

I have added a test that covers the exact scenario we are fixing. For complex issues, comment the issue number in the test for future reference.

github-actions · 2026-04-07T02:51:45Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

Copilot

Pull request overview

This PR improves ingestion agent health-check resilience by adding Resilience4j retry-with-backoff to PipelineServiceClient.getServiceStatus(), so transient pipeline-service failures (e.g., during Airflow restarts) don’t immediately mark the agent as unavailable.

Changes:

Moved retry-with-backoff logic into PipelineServiceClient.getServiceStatus() and simplified getServiceStatusBackoff() to delegate.
Updated retry to operate on PipelineServiceClientResponse (retrying when HTTP code is non-200).
Added an Airflow REST client test ensuring a transient 500 followed by 200 recovers successfully and issues the expected number of requests.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
openmetadata-spec/src/main/java/org/openmetadata/service/clients/pipeline/PipelineServiceClient.java	Centralizes retry-with-backoff in `getServiceStatus()` so all callers benefit from transient failure tolerance.
openmetadata-service/src/test/java/org/openmetadata/service/clients/pipeline/airflow/AirflowRESTClientTest.java	Adds regression test verifying recovery from a transient health-check failure.

...data-spec/src/main/java/org/openmetadata/service/clients/pipeline/PipelineServiceClient.java

...e/src/test/java/org/openmetadata/service/clients/pipeline/airflow/AirflowRESTClientTest.java

github-actions · 2026-04-07T03:31:07Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

openmetadata-mcp/src/main/java/org/openmetadata/mcp/tools/GetEntityVersionsTool.java

openmetadata-mcp/src/main/java/org/openmetadata/mcp/tools/CompareEntityVersionsTool.java

github-actions · 2026-04-07T04:54:50Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

...data-spec/src/main/java/org/openmetadata/service/clients/pipeline/PipelineServiceClient.java

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

...data-spec/src/main/java/org/openmetadata/service/clients/pipeline/PipelineServiceClient.java

github-actions · 2026-04-07T05:38:34Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

...data-spec/src/main/java/org/openmetadata/service/clients/pipeline/PipelineServiceClient.java

github-actions · 2026-04-07T05:50:20Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

...data-spec/src/main/java/org/openmetadata/service/clients/pipeline/PipelineServiceClient.java

github-actions · 2026-04-07T06:00:30Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

...data-spec/src/main/java/org/openmetadata/service/clients/pipeline/PipelineServiceClient.java

github-actions · 2026-04-07T06:12:38Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

...data-spec/src/main/java/org/openmetadata/service/clients/pipeline/PipelineServiceClient.java

… for transient failure recovery

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

Copilot · 2026-04-08T00:44:35Z

...data-spec/src/main/java/org/openmetadata/service/clients/pipeline/PipelineServiceClient.java

+  /**
+   * Check the status of pipeline service to ensure it is healthy. Retries up to {@link
+   * #MAX_ATTEMPTS} times with backoff so that a single transient failure (e.g. Airflow restart,
+   * network blip) does not permanently mark the agent as unavailable. Responses with status codes
+   * &ge; 500 are retried; known non-transient 4xx responses (401, 403, 404, version mismatch) are
+   * returned immediately. Any other unexpected HTTP status code is passed through as-is and will
+   * not trigger retries.


The Javadoc implies that auth/plugin/version mismatch cases will come back as 4xx and therefore bypass retries, but getServiceStatus() only decides retries based on response.getCode() >= 500. Implementations that translate permanent config/auth/RBAC failures into buildUnhealthyStatus(...) (code 500) will still be retried and incur backoff delays. Consider updating the Javadoc to describe the actual contract (implementations must preserve non-retryable 4xx/422 codes), and/or add a protected isRetryable(PipelineServiceClientResponse) hook so clients can explicitly mark permanent failures as non-retryable.

Copilot · 2026-04-08T00:44:35Z

...data-spec/src/main/java/org/openmetadata/service/clients/pipeline/PipelineServiceClient.java

+          var retryConfig =
+              RetryConfig.<PipelineServiceClientResponse>custom()
+                  .maxAttempts(MAX_ATTEMPTS)
+                  .waitDuration(Duration.ofMillis(getRetryBackoffMillis()))
+                  .retryOnResult(response -> response == null || response.getCode() >= 500)
+                  .failAfterMaxAttempts(false)
+                  .build();


The PR description says getServiceStatus() retries when the response code is not 200, but the current predicate only retries on response == null or code >= 500 (and will pass through 4xx/429/etc without retry). Please either update the PR description / method docs to match the actual behavior, or broaden the retry predicate to include the intended retryable non-200 statuses (e.g., 429/408) if that was the goal.

Copilot · 2026-04-08T00:44:36Z

...data-spec/src/main/java/org/openmetadata/service/clients/pipeline/PipelineServiceClient.java

  private static final Integer MAX_ATTEMPTS = 3;
-  private static final Integer BACKOFF_TIME_SECONDS = 5;
+  private static final long DEFAULT_BACKOFF_MILLIS = 5_000L;
  private static final String DISABLED_STATUS = "disabled";

+  private volatile Retry serviceStatusRetry;
+


DEFAULT_BACKOFF_MILLIS hardcodes a 5s sleep between retries, meaning getServiceStatus() can block for ~10s on repeated retryable failures. This will directly impact REST endpoints that call this method (e.g., validation/status checks) and can also slow down unit tests that exercise unhealthy paths unless they override getRetryBackoffMillis(). Consider making the backoff configurable via PipelineServiceClientConfiguration (or similar) so production can keep 5s while tests/interactive calls can use a smaller value.

…errors

gitar-bot · 2026-04-12T04:38:39Z

Code Review ✅ Approved 10 resolved / 10 findings

Adds retry-with-backoff to getServiceStatus for transient failure recovery, addressing 10 issues including null response handling, permanent error classification, and unnecessary retry delays. All findings resolved.

✅ 10 resolved

✅ Bug: GetEntityVersionsTool calls non-existent CommonUtils.parseIntParam()

📄 openmetadata-mcp/src/main/java/org/openmetadata/mcp/tools/GetEntityVersionsTool.java:51
At line 51, CommonUtils.parseIntParam(params.get("limit"), DEFAULT_LIMIT) is called, but the CommonUtils class in the MCP tools package does not define a parseIntParam() method. This will cause a compilation error.

✅ Bug: CompareEntityVersionsTool NPE when pojoToJson returns null

📄 openmetadata-mcp/src/main/java/org/openmetadata/mcp/tools/CompareEntityVersionsTool.java:115-117
JsonUtils.pojoToJson(null) returns null. At line 117, fromJson.equals(toJson) will throw a NullPointerException if either fromVal or toVal is null (e.g., a field not present in one version). This is a reachable condition since version comparisons commonly involve fields that were added or removed.

✅ Bug: CompareEntityVersionsTool hardcodes table-specific fields

📄 openmetadata-mcp/src/main/java/org/openmetadata/mcp/tools/CompareEntityVersionsTool.java:97-100
The computeDifferences method compares a hardcoded list of fields (columns, tableConstraints, tableType, etc.) that are specific to Table entities. When comparing other entity types (Pipeline, Dashboard, Topic, etc.), this tool will miss all entity-specific fields and only compare the few common ones (description, owners, tags, displayName). The tool's name and parameters suggest it is generic.

✅ Edge Case: MCP tools NPE when exception message is null

📄 openmetadata-mcp/src/main/java/org/openmetadata/mcp/tools/CompareEntityVersionsTool.java:90 📄 openmetadata-mcp/src/main/java/org/openmetadata/mcp/tools/GetEntityVersionsTool.java:72
Both CompareEntityVersionsTool (line 90) and GetEntityVersionsTool (line 72) catch all exceptions and call Map.of("error", e.getMessage()). Map.of() does not permit null values, so if e.getMessage() is null (common for NPE, ClassCastException, etc.), this will throw a NullPointerException, masking the original error.

✅ Quality: PR includes unrelated files (notes, scripts, issue templates)

📄 TOP_10_CONTRIBUTIONS.txt 📄 TOP_10_CONTRIBUTIONS_V2.txt 📄 issue-assignment-requests.md 📄 ISSUE_DESCRIPTION.md 📄 ISSUE_DESCRIPTION_RESOURCE_LEAKS.md 📄 ISSUE_SUBJECT_CONTEXT_SILENT_CATCHES.md 📄 PR_DESCRIPTION.md 📄 PR_DESCRIPTION_AIRFLOW_RETRY.md 📄 PR_DESCRIPTION_RESOURCE_LEAKS.md 📄 scripts/fix_basic.py 📄 openmetadata-mcp/src/main/java/org/openmetadata/mcp/tools/CompareEntityVersionsTool.java 📄 openmetadata-mcp/src/main/java/org/openmetadata/mcp/tools/GetEntityVersionsTool.java
The PR includes 12 files that are unrelated to the stated fix (retry-with-backoff for getServiceStatus): TOP_10_CONTRIBUTIONS.txt, TOP_10_CONTRIBUTIONS_V2.txt, issue-assignment-requests.md, ISSUE_DESCRIPTION.md, ISSUE_DESCRIPTION_RESOURCE_LEAKS.md, ISSUE_SUBJECT_CONTEXT_SILENT_CATCHES.md, PR_DESCRIPTION*.md, scripts/fix_basic.py, and the two MCP tool files. These appear to be personal working notes and unrelated feature additions that should be in separate PRs. The fix_basic.py script contains a hardcoded Windows path (D:\OpenMetadata\...).

...and 5 more resolved from earlier reviews

Options

Display: compact → Showing less information.

Comment with these commands to change:

`Compact`
`gitar display:verbose`

_{Was this helpful? React with 👍 / 👎 | Gitar}

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

sonarqubecloud · 2026-04-12T05:38:09Z

Quality Gate passed for 'open-metadata-ingestion'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

Copilot AI review requested due to automatic review settings April 7, 2026 02:51

Copilot started reviewing on behalf of RajdeepKushwaha5 April 7, 2026 02:51 View session

Copilot AI reviewed Apr 7, 2026

View reviewed changes

RajdeepKushwaha5 force-pushed the fix/25666-airflow-status-retry-on-transient-errors branch from 68791c8 to 9336512 Compare April 7, 2026 03:30

gitar-bot bot reviewed Apr 7, 2026

View reviewed changes

openmetadata-mcp/src/main/java/org/openmetadata/mcp/tools/GetEntityVersionsTool.java Outdated Show resolved Hide resolved

gitar-bot bot reviewed Apr 7, 2026

View reviewed changes

openmetadata-mcp/src/main/java/org/openmetadata/mcp/tools/CompareEntityVersionsTool.java Outdated Show resolved Hide resolved

gitar-bot bot reviewed Apr 7, 2026

View reviewed changes

openmetadata-mcp/src/main/java/org/openmetadata/mcp/tools/CompareEntityVersionsTool.java Outdated Show resolved Hide resolved

gitar-bot bot reviewed Apr 7, 2026

View reviewed changes

openmetadata-mcp/src/main/java/org/openmetadata/mcp/tools/CompareEntityVersionsTool.java Outdated Show resolved Hide resolved

Copilot AI review requested due to automatic review settings April 7, 2026 04:54

RajdeepKushwaha5 force-pushed the fix/25666-airflow-status-retry-on-transient-errors branch from 9336512 to f32ab33 Compare April 7, 2026 04:54

Copilot started reviewing on behalf of RajdeepKushwaha5 April 7, 2026 04:55 View session

gitar-bot bot reviewed Apr 7, 2026

View reviewed changes

...data-spec/src/main/java/org/openmetadata/service/clients/pipeline/PipelineServiceClient.java Outdated Show resolved Hide resolved

Copilot AI reviewed Apr 7, 2026

View reviewed changes

...data-spec/src/main/java/org/openmetadata/service/clients/pipeline/PipelineServiceClient.java Outdated Show resolved Hide resolved

...data-spec/src/main/java/org/openmetadata/service/clients/pipeline/PipelineServiceClient.java Outdated Show resolved Hide resolved

RajdeepKushwaha5 force-pushed the fix/25666-airflow-status-retry-on-transient-errors branch from f32ab33 to 0d023dc Compare April 7, 2026 05:38

gitar-bot bot reviewed Apr 7, 2026

View reviewed changes

...data-spec/src/main/java/org/openmetadata/service/clients/pipeline/PipelineServiceClient.java Outdated Show resolved Hide resolved

Copilot AI review requested due to automatic review settings April 7, 2026 05:49

RajdeepKushwaha5 force-pushed the fix/25666-airflow-status-retry-on-transient-errors branch from 0d023dc to 7a38087 Compare April 7, 2026 05:49

Copilot started reviewing on behalf of RajdeepKushwaha5 April 7, 2026 05:50 View session

Copilot AI reviewed Apr 7, 2026

View reviewed changes

...data-spec/src/main/java/org/openmetadata/service/clients/pipeline/PipelineServiceClient.java Outdated Show resolved Hide resolved

RajdeepKushwaha5 force-pushed the fix/25666-airflow-status-retry-on-transient-errors branch from 7a38087 to a6412ae Compare April 7, 2026 06:00

gitar-bot bot reviewed Apr 7, 2026

View reviewed changes

...data-spec/src/main/java/org/openmetadata/service/clients/pipeline/PipelineServiceClient.java Outdated Show resolved Hide resolved

Copilot AI review requested due to automatic review settings April 7, 2026 06:12

RajdeepKushwaha5 force-pushed the fix/25666-airflow-status-retry-on-transient-errors branch from a6412ae to 2676e07 Compare April 7, 2026 06:12

RajdeepKushwaha5 had a problem deploying to test April 8, 2026 00:22 — with GitHub Actions Error

RajdeepKushwaha5 force-pushed the fix/25666-airflow-status-retry-on-transient-errors branch from ecd31ba to f33984c Compare April 8, 2026 00:32

gitar-bot bot reviewed Apr 8, 2026

View reviewed changes

...data-spec/src/main/java/org/openmetadata/service/clients/pipeline/PipelineServiceClient.java Outdated Show resolved Hide resolved

Fixes open-metadata#25666: Add retry-with-backoff to getServiceStatus…

b0f9406

… for transient failure recovery

Copilot AI review requested due to automatic review settings April 8, 2026 00:39

RajdeepKushwaha5 force-pushed the fix/25666-airflow-status-retry-on-transient-errors branch from f33984c to b0f9406 Compare April 8, 2026 00:39

Copilot started reviewing on behalf of RajdeepKushwaha5 April 8, 2026 00:40 View session

Copilot AI reviewed Apr 8, 2026

View reviewed changes

RajdeepKushwaha5 temporarily deployed to test April 8, 2026 00:50 — with GitHub Actions Inactive

Merge branch 'main' into fix/25666-airflow-status-retry-on-transient-…

0b675f3

…errors

RajdeepKushwaha5 requested a review from Copilot April 12, 2026 04:39

Copilot started reviewing on behalf of RajdeepKushwaha5 April 12, 2026 04:39 View session

Copilot AI reviewed Apr 12, 2026

View reviewed changes

RajdeepKushwaha5 temporarily deployed to test April 12, 2026 04:47 — with GitHub Actions Inactive

Conversation

RajdeepKushwaha5 commented Apr 7, 2026

Describe your changes:

Type of change:

Checklist:

Uh oh!

github-actions bot commented Apr 7, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Apr 7, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Apr 7, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Apr 7, 2026

Uh oh!

Uh oh!

github-actions bot commented Apr 7, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

github-actions bot commented Apr 7, 2026

Uh oh!

Uh oh!

github-actions bot commented Apr 7, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

gitar-bot bot commented Apr 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

sonarqubecloud bot commented Apr 12, 2026

Quality Gate passed for 'open-metadata-ingestion'

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gitar-bot bot commented Apr 12, 2026 •

edited

Loading