Skip to content

Implement observability and circuit breaker for identity service#1653

Open
SurbhiAgarwal1 wants to merge 1 commit into
hyperledger-labs:mainfrom
SurbhiAgarwal1:pr-1600
Open

Implement observability and circuit breaker for identity service#1653
SurbhiAgarwal1 wants to merge 1 commit into
hyperledger-labs:mainfrom
SurbhiAgarwal1:pr-1600

Conversation

@SurbhiAgarwal1
Copy link
Copy Markdown
Contributor

Description

This PR implements observability (metrics) and a lightweight, in-memory circuit breaker for the identity service in token/services/identity. These enhancements improve the resilience and monitorability of the service under load and failure conditions.

Fixes: #1644

Changes

1. Metrics Instrumentation

  • Defined IdentityMetrics struct in token/services/identity/metrics.go.
  • Added support for the following metrics via metrics.Provider:
    • identity_requests_total (Counter): Total number of identity requests.
    • identity_errors_total (Counter): Total number of identity errors.
    • identity_request_latency_ms (Histogram): Request latency in milliseconds.
    • identity_inflight_requests (Gauge): Number of concurrent requests.
  • Integrated metrics collection into all public entry points of the identity.Provider. Metrics are collected at the start of each call to ensure even rejected requests are tracked.

2. Circuit Breaker

  • Implemented a lightweight CircuitBreaker in token/services/identity/cb.go.
  • Supports configurable failure thresholds and cooldown periods.
  • Automatically rejects requests with a "back-pressure" error when the circuit is open.
  • Resets failure counts and closes the circuit after a successful operation or cooldown period.

3. Provider Integration

  • Updated identity.Provider to support optional metrics and circuit breaker initialization via functional options (WithMetrics, WithCircuitBreaker).
  • Wrapped core logic with the circuit breaker Allow() check and failure/success recording.
  • Ensured zero impact on existing business logic and maintained full compatibility with the driver.IdentityProvider interface.

4. Testing

  • Added token/services/identity/observability_test.go with unit tests covering:
    • Basic metrics increment sanity.
    • Circuit breaker opening after the failure threshold is reached.
    • Request rejection during the open state.
    • Circuit reset behavior after the cooldown period.

Verification Results

All tests in the identity service package passed successfully:

=== RUN   TestCircuitBreaker
--- PASS: TestCircuitBreaker (0.15s)
=== RUN   TestProviderObservability
--- PASS: TestProviderObservability (0.00s)
=== RUN   TestProvider_RegisterRecipientData
--- PASS: TestProvider_RegisterRecipientData (0.00s)
...
PASS
ok  github.com/hyperledger-labs/fabric-token-sdk/token/services/identity

@AkramBitar
Copy link
Copy Markdown
Contributor

@SurbhiAgarwal1 thanks a lot for openning this PR.
Please like the GIT issue to this PR.
If we do not have one please create one for this PR with description about the problem.

@adecaro
Copy link
Copy Markdown
Contributor

adecaro commented May 5, 2026

Hi @SurbhiAgarwal1 , I think you introduced in this PR changes concerning auditable identities that are not relevant here. Please, revert those changes. Thanks 🙏

…erledger-labs#1644)

Signed-off-by: SurbhiAgarwal1 <agarwalsurbhi1807@gmail.com>
@SurbhiAgarwal1
Copy link
Copy Markdown
Contributor Author

SurbhiAgarwal1 commented May 5, 2026

Hey @adecaro @AkramBitar, I’ve cleaned up the history as you suggested. Would appreciate a look when you have a moment! 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Observability, Alerting, and Circuit Breaking [LOW]

3 participants