Circuit Breaker Pattern for Service Resilience

## Summary
Implement the Circuit Breaker pattern to provide automatic failure detection, fast-fail behavior, and service resilience when dealing with unreliable external services.

## Motivation
The Circuit Breaker pattern is essential for building resilient distributed systems:

- **Failure Isolation**: Prevent cascading failures when downstream services are unavailable
- **Fast Fail**: Avoid wasting resources on requests likely to fail
- **Service Recovery**: Automatically detect when failed services recover
- **User Experience**: Provide immediate feedback instead of long timeouts
- **System Stability**: Prevent resource exhaustion during service outages
- **Graceful Degradation**: Enable fallback mechanisms when services are down
- **Operational Visibility**: Monitor service health and failure patterns
- **Cost Optimization**: Reduce unnecessary API calls and timeouts

Without circuit breakers, applications can become unresponsive when dependencies fail, leading to poor user experience and potential system-wide failures.

## Proposed API

```php
// Basic circuit breaker
$client = fetch_client()
    ->withCircuitBreaker([
        'failure_threshold' => 5,        // Trip after 5 failures
        'timeout' => 60,                 // Stay open for 60 seconds
        'recovery_timeout' => 30,        // Half-open timeout
        'success_threshold' => 3,        // Require 3 successes to close
    ])
    ->get('/unreliable-service');

// Advanced configuration
$client = fetch_client()
    ->withCircuitBreaker([
        'name' => 'payment-service',
        'failure_threshold' => 10,
        'failure_rate_threshold' => 0.5,  // 50% failure rate
        'minimum_requests' => 20,         // Minimum requests before calculating rate
        'timeout' => 300,                 // 5 minutes open
        'recovery_timeout' => 60,         // 1 minute half-open
        'success_threshold' => 5,
        'failure_criteria' => [
            'status_codes' => [500, 502, 503, 504],
            'exceptions' => [ConnectException::class, TimeoutException::class],
        ],
        'metrics_window' => 300,          // 5-minute sliding window
        'fallback' => fn() => $this->getDefaultPaymentResponse(),
    ]);

// Per-service circuit breakers
$paymentBreaker = new CircuitBreaker('payment-service', [
    'failure_threshold' => 5,
    'timeout' => 120,
]);

$inventoryBreaker = new CircuitBreaker('inventory-service', [
    'failure_threshold' => 3,
    'timeout' => 60,
]);

$client = fetch_client()
    ->withCircuitBreaker($paymentBreaker)
    ->post('/payment/process', $paymentData);

// Circuit breaker with fallback strategies
fetch('/api/recommendations')
    ->withCircuitBreaker([
        'name' => 'recommendations',
        'fallback_chain' => [
            new CachedFallback($cache, 'recommendations'),
            new DefaultValueFallback(['popular_items' => []]),
            new ThrowExceptionFallback(new ServiceUnavailableException()),
        ],
    ]);
```

## Implementation Details

### Circuit Breaker States
```php
enum CircuitBreakerState: string
{
    case CLOSED = 'closed';       // Normal operation
    case OPEN = 'open';           // Failing fast
    case HALF_OPEN = 'half_open'; // Testing recovery
}

class CircuitBreakerException extends RuntimeException
{
    public function __construct(
        string $message,
        private string $serviceName,
        private CircuitBreakerState $state,
        private array $metrics = []
    ) {
        parent::__construct($message);
    }
    
    public function getServiceName(): string { return $this->serviceName; }
    public function getState(): CircuitBreakerState { return $this->state; }
    public function getMetrics(): array { return $this->metrics; }
}
```

### Core Circuit Breaker Implementation
```php
interface CircuitBreakerInterface
{
    public function call(callable $operation): mixed;
    public function getState(): CircuitBreakerState;
    public function getMetrics(): CircuitBreakerMetrics;
    public function forceOpen(): void;
    public function forceClosed(): void;
    public function reset(): void;
}

class CircuitBreaker implements CircuitBreakerInterface
{
    private CircuitBreakerState $state = CircuitBreakerState::CLOSED;
    private CircuitBreakerMetrics $metrics;
    private CircuitBreakerConfiguration $config;
    private float $lastFailureTime = 0;
    private int $consecutiveSuccesses = 0;
    
    public function __construct(
        private string $name,
        array $config = []
    ) {
        $this->config = new CircuitBreakerConfiguration($config);
        $this->metrics = new CircuitBreakerMetrics($this->config->getMetricsWindow());
    }
    
    public function call(callable $operation): mixed
    {
        // Check if circuit should trip
        $this->updateState();
        
        switch ($this->state) {
            case CircuitBreakerState::OPEN:
                return $this->handleOpenState();
                
            case CircuitBreakerState::HALF_OPEN:
                return $this->handleHalfOpenState($operation);
                
            case CircuitBreakerState::CLOSED:
                return $this->handleClosedState($operation);
        }
    }
    
    private function handleOpenState(): mixed
    {
        $this->metrics->recordFastFail();
        
        if ($this->config->hasFallback()) {
            return $this->config->getFallback()();
        }
        
        throw new CircuitBreakerException(
            "Circuit breaker '{$this->name}' is OPEN",
            $this->name,
            $this->state,
            $this->metrics->toArray()
        );
    }
    
    private function handleHalfOpenState(callable $operation): mixed
    {
        try {
            $result = $operation();
            $this->onSuccess();
            return $result;
        } catch (\Throwable $e) {
            $this->onFailure($e);
            throw $e;
        }
    }
    
    private function handleClosedState(callable $operation): mixed
    {
        try {
            $result = $operation();
            $this->onSuccess();
            return $result;
        } catch (\Throwable $e) {
            if ($this->isFailureCriteria($e)) {
                $this->onFailure($e);
            }
            throw $e;
        }
    }
    
    private function onSuccess(): void
    {
        $this->metrics->recordSuccess();
        $this->consecutiveSuccesses++;
        
        if ($this->state === CircuitBreakerState::HALF_OPEN) {
            if ($this->consecutiveSuccesses >= $this->config->getSuccessThreshold()) {
                $this->state = CircuitBreakerState::CLOSED;
                $this->consecutiveSuccesses = 0;
                $this->emitStateChange();
            }
        }
    }
    
    private function onFailure(\Throwable $exception): void
    {
        $this->metrics->recordFailure($exception);
        $this->lastFailureTime = microtime(true);
        $this->consecutiveSuccesses = 0;
        
        if ($this->shouldTrip()) {
            $this->state = CircuitBreakerState::OPEN;
            $this->emitStateChange();
        }
    }
    
    private function shouldTrip(): bool
    {
        // Check consecutive failure threshold
        if ($this->metrics->getConsecutiveFailures() >= $this->config->getFailureThreshold()) {
            return true;
        }
        
        // Check failure rate threshold
        if ($this->metrics->getTotalRequests() >= $this->config->getMinimumRequests()) {
            $failureRate = $this->metrics->getFailureRate();
            if ($failureRate >= $this->config->getFailureRateThreshold()) {
                return true;
            }
        }
        
        return false;
    }
    
    private function updateState(): void
    {
        if ($this->state === CircuitBreakerState::OPEN) {
            $timeoutElapsed = (microtime(true) - $this->lastFailureTime) >= $this->config->getTimeout();
            
            if ($timeoutElapsed) {
                $this->state = CircuitBreakerState::HALF_OPEN;
                $this->emitStateChange();
            }
        }
    }
    
    private function isFailureCriteria(\Throwable $exception): bool
    {
        // Check exception types
        foreach ($this->config->getFailureExceptions() as $exceptionClass) {
            if ($exception instanceof $exceptionClass) {
                return true;
            }
        }
        
        // Check HTTP status codes if it's an HTTP exception
        if ($exception instanceof HttpException) {
            $statusCode = $exception->getResponse()->getStatusCode();
            return in_array($statusCode, $this->config->getFailureStatusCodes());
        }
        
        return false;
    }
    
    private function emitStateChange(): void
    {
        // Emit event for monitoring
        EventManager::emit(new CircuitBreakerStateChangedEvent(
            $this->name,
            $this->state,
            $this->metrics->toArray()
        ));
    }
}
```

### Circuit Breaker Metrics
```php
class CircuitBreakerMetrics
{
    private SlidingWindowCounter $requests;
    private SlidingWindowCounter $failures;
    private SlidingWindowCounter $successes;
    private int $consecutiveFailures = 0;
    private int $fastFails = 0;
    private float $lastRequestTime = 0;
    
    public function __construct(int $windowSize = 300)
    {
        $this->requests = new SlidingWindowCounter($windowSize);
        $this->failures = new SlidingWindowCounter($windowSize);
        $this->successes = new SlidingWindowCounter($windowSize);
    }
    
    public function recordSuccess(): void
    {
        $this->requests->increment();
        $this->successes->increment();
        $this->consecutiveFailures = 0;
        $this->lastRequestTime = microtime(true);
    }
    
    public function recordFailure(\Throwable $exception): void
    {
        $this->requests->increment();
        $this->failures->increment();
        $this->consecutiveFailures++;
        $this->lastRequestTime = microtime(true);
    }
    
    public function recordFastFail(): void
    {
        $this->fastFails++;
    }
    
    public function getFailureRate(): float
    {
        $total = $this->requests->getCount();
        return $total > 0 ? $this->failures->getCount() / $total : 0;
    }
    
    public function getConsecutiveFailures(): int
    {
        return $this->consecutiveFailures;
    }
    
    public function getTotalRequests(): int
    {
        return $this->requests->getCount();
    }
    
    public function toArray(): array
    {
        return [
            'total_requests' => $this->getTotalRequests(),
            'successful_requests' => $this->successes->getCount(),
            'failed_requests' => $this->failures->getCount(),
            'consecutive_failures' => $this->consecutiveFailures,
            'failure_rate' => $this->getFailureRate(),
            'fast_fails' => $this->fastFails,
            'last_request_time' => $this->lastRequestTime,
        ];
    }
}

class SlidingWindowCounter
{
    private array $buckets = [];
    private int $windowSize;
    private int $bucketSize = 10; // 10-second buckets
    
    public function __construct(int $windowSize)
    {
        $this->windowSize = $windowSize;
    }
    
    public function increment(): void
    {
        $bucket = $this->getCurrentBucket();
        $this->buckets[$bucket] = ($this->buckets[$bucket] ?? 0) + 1;
        $this->cleanup();
    }
    
    public function getCount(): int
    {
        $this->cleanup();
        return array_sum($this->buckets);
    }
    
    private function getCurrentBucket(): int
    {
        return (int) (time() / $this->bucketSize);
    }
    
    private function cleanup(): void
    {
        $cutoff = $this->getCurrentBucket() - ($this->windowSize / $this->bucketSize);
        $this->buckets = array_filter(
            $this->buckets,
            fn($bucket) => $bucket > $cutoff,
            ARRAY_FILTER_USE_KEY
        );
    }
}
```

### Fallback Strategies
```php
interface FallbackInterface
{
    public function execute(): mixed;
    public function canHandle(\Throwable $exception): bool;
}

class CachedFallback implements FallbackInterface
{
    public function __construct(
        private CacheInterface $cache,
        private string $cacheKey,
        private int $maxAge = 3600
    ) {}
    
    public function execute(): mixed
    {
        $cached = $this->cache->get($this->cacheKey);
        
        if ($cached && $this->isFresh($cached)) {
            return $cached['data'];
        }
        
        throw new FallbackException('No cached data available');
    }
    
    public function canHandle(\Throwable $exception): bool
    {
        return $exception instanceof CircuitBreakerException;
    }
    
    private function isFresh(array $cached): bool
    {
        return (time() - $cached['timestamp']) <= $this->maxAge;
    }
}

class DefaultValueFallback implements FallbackInterface
{
    public function __construct(private mixed $defaultValue) {}
    
    public function execute(): mixed
    {
        return $this->defaultValue;
    }
    
    public function canHandle(\Throwable $exception): bool
    {
        return true;
    }
}

class FallbackChain implements FallbackInterface
{
    public function __construct(private array $fallbacks) {}
    
    public function execute(): mixed
    {
        $lastException = null;
        
        foreach ($this->fallbacks as $fallback) {
            try {
                return $fallback->execute();
            } catch (\Throwable $e) {
                $lastException = $e;
                continue;
            }
        }
        
        throw $lastException ?? new FallbackException('All fallbacks failed');
    }
    
    public function canHandle(\Throwable $exception): bool
    {
        return true;
    }
}
```

### Circuit Breaker Registry
```php
class CircuitBreakerRegistry
{
    private static array $breakers = [];
    
    public static function get(string $name, array $config = []): CircuitBreaker
    {
        if (!isset(self::$breakers[$name])) {
            self::$breakers[$name] = new CircuitBreaker($name, $config);
        }
        
        return self::$breakers[$name];
    }
    
    public static function getAll(): array
    {
        return self::$breakers;
    }
    
    public static function getMetrics(): array
    {
        $metrics = [];
        
        foreach (self::$breakers as $name => $breaker) {
            $metrics[$name] = [
                'state' => $breaker->getState()->value,
                'metrics' => $breaker->getMetrics()->toArray(),
            ];
        }
        
        return $metrics;
    }
    
    public static function reset(string $name = null): void
    {
        if ($name) {
            self::$breakers[$name]?->reset();
        } else {
            foreach (self::$breakers as $breaker) {
                $breaker->reset();
            }
        }
    }
}
```

### Integration with Fetch Client
```php
trait SupportsCircuitBreaker
{
    private ?CircuitBreaker $circuitBreaker = null;
    
    public function withCircuitBreaker(CircuitBreaker|array $breaker): self
    {
        if (is_array($breaker)) {
            $name = $breaker['name'] ?? 'default';
            $this->circuitBreaker = CircuitBreakerRegistry::get($name, $breaker);
        } else {
            $this->circuitBreaker = $breaker;
        }
        
        return $this;
    }
    
    protected function executeWithCircuitBreaker(callable $operation): mixed
    {
        if (!$this->circuitBreaker) {
            return $operation();
        }
        
        return $this->circuitBreaker->call($operation);
    }
    
    public function sendRequest(string $method, string $uri, array $options = []): ResponseInterface|PromiseInterface
    {
        return $this->executeWithCircuitBreaker(function () use ($method, $uri, $options) {
            return parent::sendRequest($method, $uri, $options);
        });
    }
}
```

## Advanced Features

### Adaptive Circuit Breaker
```php
class AdaptiveCircuitBreaker extends CircuitBreaker
{
    private float $baseFailureThreshold;
    private MovingAverage $responseTimeAverage;
    
    public function __construct(string $name, array $config = [])
    {
        parent::__construct($name, $config);
        $this->baseFailureThreshold = $this->config->getFailureThreshold();
        $this->responseTimeAverage = new MovingAverage(100);
    }
    
    protected function shouldTrip(): bool
    {
        // Adjust threshold based on response time trends
        $avgResponseTime = $this->responseTimeAverage->getAverage();
        $threshold = $this->calculateAdaptiveThreshold($avgResponseTime);
        
        $this->config->setFailureThreshold($threshold);
        
        return parent::shouldTrip();
    }
    
    private function calculateAdaptiveThreshold(float $avgResponseTime): int
    {
        // Lower threshold if response times are increasing
        $multiplier = max(0.5, min(2.0, 1.0 / (1.0 + $avgResponseTime / 1000)));
        return (int) ($this->baseFailureThreshold * $multiplier);
    }
}
```

### Circuit Breaker Health Check
```php
class CircuitBreakerHealthCheck
{
    public function __construct(private CircuitBreakerRegistry $registry) {}
    
    public function getHealthStatus(): array
    {
        $status = [
            'status' => 'healthy',
            'circuit_breakers' => [],
            'summary' => [
                'total' => 0,
                'closed' => 0,
                'open' => 0,
                'half_open' => 0,
            ],
        ];
        
        foreach ($this->registry->getAll() as $name => $breaker) {
            $state = $breaker->getState();
            $metrics = $breaker->getMetrics();
            
            $status['circuit_breakers'][$name] = [
                'state' => $state->value,
                'metrics' => $metrics->toArray(),
                'health' => $this->calculateHealth($state, $metrics),
            ];
            
            $status['summary']['total']++;
            $status['summary'][strtolower($state->value)]++;
            
            if ($state === CircuitBreakerState::OPEN) {
                $status['status'] = 'degraded';
            }
        }
        
        return $status;
    }
    
    private function calculateHealth(CircuitBreakerState $state, CircuitBreakerMetrics $metrics): string
    {
        if ($state === CircuitBreakerState::OPEN) {
            return 'unhealthy';
        }
        
        $failureRate = $metrics->getFailureRate();
        
        if ($failureRate > 0.5) {
            return 'warning';
        }
        
        return 'healthy';
    }
}
```

## Use Cases

### Microservices Architecture
```php
// Service-specific circuit breakers
$userService = fetch_client()
    ->withCircuitBreaker([
        'name' => 'user-service',
        'failure_threshold' => 5,
        'timeout' => 60,
        'fallback' => fn() => $this->getUserFromCache(),
    ]);

$paymentService = fetch_client()
    ->withCircuitBreaker([
        'name' => 'payment-service',
        'failure_threshold' => 3,
        'timeout' => 120,
        'fallback' => fn() => throw new PaymentUnavailableException(),
    ]);
```

### API Gateway Pattern
```php
class ApiGateway
{
    private array $serviceBreakers = [];
    
    public function __construct()
    {
        $this->serviceBreakers = [
            'auth' => new CircuitBreaker('auth-service', ['failure_threshold' => 3]),
            'billing' => new CircuitBreaker('billing-service', ['failure_threshold' => 5]),
            'inventory' => new CircuitBreaker('inventory-service', ['failure_threshold' => 10]),
        ];
    }
    
    public function routeRequest(string $service, Request $request): Response
    {
        $breaker = $this->serviceBreakers[$service] ?? null;
        
        if (!$breaker) {
            throw new ServiceNotFound("Service '$service' not found");
        }
        
        return $breaker->call(function () use ($service, $request) {
            return $this->forwardToService($service, $request);
        });
    }
}
```

### External API Integration
```php
// Third-party API with circuit breaker
$weatherService = fetch_client()
    ->withCircuitBreaker([
        'name' => 'weather-api',
        'failure_threshold' => 10,
        'failure_rate_threshold' => 0.3,
        'timeout' => 300,
        'fallback' => fn() => $this->getWeatherFromCache(),
    ]);

try {
    $weather = $weatherService->get('/weather/current')->json();
} catch (CircuitBreakerException $e) {
    // Service is down, use fallback
    $weather = $this->getDefaultWeather();
}
```

## Monitoring and Observability

### Circuit Breaker Dashboard
```php
class CircuitBreakerDashboard
{
    public function getDashboardData(): array
    {
        $registry = new CircuitBreakerRegistry();
        $healthCheck = new CircuitBreakerHealthCheck($registry);
        
        return [
            'health' => $healthCheck->getHealthStatus(),
            'metrics' => $registry->getMetrics(),
            'events' => $this->getRecentEvents(),
        ];
    }
    
    private function getRecentEvents(): array
    {
        // Return recent circuit breaker state changes
        return EventStore::getEvents('CircuitBreakerStateChanged', [
            'since' => time() - 3600, // Last hour
            'limit' => 100,
        ]);
    }
}
```

### Metrics Integration
```php
$client = fetch_client()
    ->withCircuitBreaker([
        'name' => 'external-api',
        'metrics_callback' => function (CircuitBreakerMetrics $metrics, CircuitBreakerState $state) {
            // Send metrics to monitoring system
            $this->metrics->gauge('circuit_breaker.failure_rate', $metrics->getFailureRate(), [
                'service' => 'external-api',
                'state' => $state->value,
            ]);
        },
    ]);
```

## Testing

```php
class CircuitBreakerTest extends TestCase
{
    public function test_circuit_breaker_trips_after_failures()
    {
        $breaker = new CircuitBreaker('test', ['failure_threshold' => 3]);
        
        // Generate failures
        for ($i = 0; $i < 3; $i++) {
            try {
                $breaker->call(fn() => throw new \Exception('Failure'));
            } catch (\Exception $e) {
                // Expected
            }
        }
        
        $this->assertEquals(CircuitBreakerState::OPEN, $breaker->getState());
        
        // Next call should fast fail
        $this->expectException(CircuitBreakerException::class);
        $breaker->call(fn() => 'success');
    }
    
    public function test_circuit_breaker_recovers_after_timeout()
    {
        $breaker = new CircuitBreaker('test', [
            'failure_threshold' => 1,
            'timeout' => 0.1, // 100ms
            'success_threshold' => 1,
        ]);
        
        // Trip the breaker
        try {
            $breaker->call(fn() => throw new \Exception('Failure'));
        } catch (\Exception $e) {}
        
        $this->assertEquals(CircuitBreakerState::OPEN, $breaker->getState());
        
        // Wait for timeout
        usleep(150000); // 150ms
        
        // Should now be half-open and accept one success
        $result = $breaker->call(fn() => 'success');
        
        $this->assertEquals('success', $result);
        $this->assertEquals(CircuitBreakerState::CLOSED, $breaker->getState());
    }
}
```

## Benefits

- **System Resilience**: Prevent cascading failures in distributed systems
- **Fast Failure**: Immediate response instead of hanging requests
- **Resource Protection**: Avoid wasting resources on failing services
- **Service Discovery**: Automatic detection of service recovery
- **Operational Insight**: Real-time visibility into service health
- **User Experience**: Graceful degradation with fallback responses
- **Cost Optimization**: Reduce unnecessary API calls and timeouts

## Priority
**High Impact, Medium Effort** - Essential for production systems dealing with external dependencies and microservices.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Circuit Breaker Pattern for Service Resilience #44

Summary

Motivation

Proposed API

Implementation Details

Circuit Breaker States

Core Circuit Breaker Implementation

Circuit Breaker Metrics

Fallback Strategies

Circuit Breaker Registry

Integration with Fetch Client

Advanced Features

Adaptive Circuit Breaker

Circuit Breaker Health Check

Use Cases

Microservices Architecture

API Gateway Pattern

External API Integration

Monitoring and Observability

Circuit Breaker Dashboard

Metrics Integration

Testing

Benefits

Priority

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Uh oh!

Circuit Breaker Pattern for Service Resilience #44

Description

Summary

Motivation

Proposed API

Implementation Details

Circuit Breaker States

Core Circuit Breaker Implementation

Circuit Breaker Metrics

Fallback Strategies

Circuit Breaker Registry

Integration with Fetch Client

Advanced Features

Adaptive Circuit Breaker

Circuit Breaker Health Check

Use Cases

Microservices Architecture

API Gateway Pattern

External API Integration

Monitoring and Observability

Circuit Breaker Dashboard

Metrics Integration

Testing

Benefits

Priority

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions