-
Notifications
You must be signed in to change notification settings - Fork 0
EN_IT_Reliability
One-line answer: A Circuit Breaker blocks calls to a failing service to prevent cascading failures, while Rate Limiting caps request volume to prevent overload and abuse — both defensive patterns that protect system resilience.
Circuit Breaker and Rate Limiting are two of the most common reliability patterns for preventing failures from propagating and overload from collapsing a system. A circuit breaker focuses on "blocking calls to a broken dependency," while rate limiting focuses on "controlling the volume of requests itself." Both patterns stop a problem in one component from spreading into a cascading failure across a distributed system.
The circuit breaker takes its name from an electrical breaker. It wraps calls to a dependent service, and when consecutive failures exceed a defined threshold, it "opens" the circuit—no longer sending calls and instead failing immediately. This avoids piling more load onto a struggling service, and lets the caller fail fast instead of waiting on timeouts.
| State | Meaning | Behavior |
|---|---|---|
| Closed | Normal state | All calls pass through. Failure rate/count is tracked |
| Open | Blocked state | Calls are rejected immediately (fail-fast). No requests reach the real dependency |
| Half-Open | Probing state | After a wait period, only a few trial requests are allowed to check whether the dependency has recovered |
- Closed → Open: When the failure threshold is exceeded. e.g. "50%+ of the last 20 requests failed" or "5 consecutive failures"
- Open → Half-Open: After a wait period (e.g. 30s) following the open, it automatically attempts a recovery probe
- Half-Open → Closed: If trial requests succeed, return to normal and reset the counter
- Half-Open → Open: If a trial request fails, re-open and reset the wait timer
This is the criterion that decides when to open the circuit. A failure-rate-based trigger is more common than a raw count, and it is paired with a minimum number of calls so the decision is made only after a sufficient sample. Too sensitive, and transient blips trip the breaker; too dull, and it trips too late to stop a cascade.
Named after a ship's bulkheads. It isolates the resources allocated to a dependency (thread pool, connections, concurrent call limit) so that a failure in one dependency cannot exhaust all resources. For example, separating the thread pool used to call service A from the one used to call service B means that even if A slows down, calls to B are unaffected. Combined with a circuit breaker, you get the double defense of "block + isolate."
- resilience4j: The de-facto standard today (Java). Provides Circuit Breaker, Rate Limiter, Bulkhead, Retry, and Time Limiter as functional decorators
- Hystrix (legacy): Netflix's original circuit breaker library. Entered maintenance mode in 2018, so it is no longer recommended for new adoption (superseded by resilience4j)
- Envoy / Istio: Provides outlier-detection-based circuit breaking at the service-mesh level with no code changes
Limiting requests to a certain rate protects the backend and ensures fairness. The main algorithms are below.
| Algorithm | How it works | Allows bursts | Characteristics |
|---|---|---|---|
| Token Bucket | Tokens refill at a steady rate; each request consumes one token. No token, request rejected | Yes (up to the bucket's accumulated tokens) | Average-rate limit plus short bursts. The most widely used |
| Leaky Bucket | Requests enter a queue (bucket) and drain (leak) at a fixed rate | No (fixed output rate) | Smoothly flattens output. Good for traffic shaping |
| Fixed Window | Counter resets each fixed window (e.g. 1 minute), allows N requests per window | Spikes at window boundaries | Simplest to implement. Suffers a 2x-burst problem near boundaries |
| Sliding Window | Counts against a moving time window (log-based or weighted approximation) | Smoothly limited | Solves the Fixed Window boundary problem. Higher implementation/memory cost |
Because Fixed Window resets its counter the instant the window rolls over, if requests cluster at the end of one window and the start of the next, nearly twice the limit can pass in a short span. Sliding Window continuously tracks the most recent N seconds to mitigate this.
- API Gateway: Applied per client/API key in Kong, NGINX, AWS API Gateway, Apigee, etc. The most common location
- Service Mesh: Istio/Envoy local rate limit and global rate limit (integrating with an external rate-limit service)
- Application level: resilience4j RateLimiter, Redis-based distributed counters (a shared limit across many instances)
One-line answer: The property that performing the same operation multiple times yields the same result as performing it once — the core of API design that stays safe under retries and duplicate requests (e.g. PUT, idempotency keys).
Idempotency is the property that performing the same operation once or many times leaves the system in the same final state. In distributed systems the network is inherently unreliable, so retries are unavoidable—and only with idempotency can you retry safely. The key idea is "sending it again has no side effect."
| Method | Idempotent | Description |
|---|---|---|
| GET | Yes | Read-only, so any number of calls changes nothing. Also Safe |
| PUT | Yes | "Overwrites" a resource to a specific state. N PUTs yield the same result |
| DELETE | Yes | Deleting an already-deleted resource still leaves a "not-present" state. (The response code may differ) |
| POST | No | Usually "creates a new resource." Two calls may create two |
| PATCH | It depends | Setting an absolute value is idempotent; an increment (e.g. +1) is not |
POST is inherently non-idempotent, but an Idempotency Key makes safe retries possible. The client sends a unique key (e.g. a UUID) per request in a header, and the server stores the processing result against that key. When a retry arrives with the same key, the server does not re-execute the operation and instead returns the stored result.
POST /payments
Idempotency-Key: 3f1a9c2e-...-b7
1st request → execute payment, store result with the key
(client retries due to a network timeout)
2nd request (same key) → return the stored result without re-executing
Payment APIs such as Stripe and PayPal adopt this pattern as a standard. You must also design the key's retention (TTL) and concurrency handling (concurrent requests with the same key).
Delivery guarantees in message queues (Kafka, SQS, etc.) are usually one of three:
- At-Most-Once: No duplicates, but messages may be lost
- At-Least-Once: No loss, but duplicates possible—the most common practical choice
- Exactly-Once: Ideal, but extremely hard to fully guarantee end-to-end
In practice, the combination of "At-Least-Once delivery + idempotent processing on the consumer side" produces an effective Exactly-Once result. Pure distributed Exactly-Once is closer to an "illusion," and idempotency is the means that realizes that illusion in practice.
A technique that filters duplicate messages/requests so they are processed only once, even when the same one arrives twice.
- By message ID: Record processed message IDs in a store (e.g. Redis) and skip any ID already seen
- By content hash: Judge identity by hashing the payload
- Built-in queue features: SQS FIFO dedup IDs, Kafka's idempotent producer, etc.
One-line answer: Graceful Shutdown finishes in-flight requests and cleans up before terminating on a shutdown signal, while Retry/Backoff retries on failure with increasing waits (exponential backoff + jitter) to absorb transient failures.
Graceful Shutdown and Retry/Backoff prevent request loss on both sides—"shutting down cleanly" and "recovering from failure." At shutdown, finish in-flight requests; on failure, retry without overwhelming the system.
- SIGTERM (15): The polite "clean up and exit" signal. The process can catch it, finish in-flight work, and have time to release resources
- SIGKILL (9): "Terminate immediately by force." The process cannot trap or ignore it; it dies at once with no cleanup
The correct shutdown flow is to send SIGTERM first to grant a grace period, then SIGKILL to force-terminate if it has not exited within that period.
On receiving a shutdown signal, the process (1) stops accepting new requests, (2) finishes the in-flight requests it already received, and (3) removes itself from the load balancer/service discovery. This way, requests in progress are not severed.
Kubernetes follows this order when terminating a Pod. The key is to flip the readiness probe to NotReady within terminationGracePeriodSeconds (default 30s) and use a preStop hook to buy time for traffic to drain.
spec:
terminationGracePeriodSeconds: 60
containers:
- name: app
lifecycle:
preStop:
exec:
# Buy time for the endpoint to drain and finish in-flight requests
command: ["sh", "-c", "sleep 10"]
readinessProbe:
httpGet:
path: /healthz/ready
port: 8080Termination sequence:
- The Pod enters Terminating, and removal from Endpoints begins (readiness flips to NotReady)
-
Simultaneously the
preStophook runs → wait so the load balancer stops sending new traffic - SIGTERM is delivered to the container → the app finishes in-flight requests and exits
- If
terminationGracePeriodSecondsis exceeded, SIGKILL force-terminates it
If you retry immediately at a fixed interval on failure, the moment a transient outage recovers, all clients rush in at once and bring the backend down again. This simultaneous surge is called the Thundering Herd or a Retry Storm.
Increasing the retry interval exponentially (e.g. 1s → 2s → 4s → 8s) spreads out the load. But backoff alone still leaves the synchronization problem of all clients retrying "at the same instant."
Adding randomness to the backoff scatters the retry timing. It is the decisive technique for dispersing a synchronized retry herd.
-
Full Jitter:
sleep = random(0, base * 2^attempt)— fully random from 0 to the cap. Recommended by AWS -
Equal Jitter:
sleep = base * 2^attempt / 2 + random(0, base * 2^attempt / 2)— half fixed, half random
# Exponential Backoff with Full Jitter (pseudocode)
base = 0.1s, cap = 30s
for attempt in 0..maxRetries:
try: return call()
except retryable:
backoff = min(cap, base * 2 ** attempt)
sleep(random(0, backoff)) # Full Jitter
raise GiveUp
Retrying without limit becomes load in itself. Cap the share of total traffic that retries may consume (e.g. "allow retries for up to 10% of requests") as a budget, preventing retries from exploding during an outage.
- Timeout: The maximum time a single call waits for a response. Prevents indefinite waiting
- Deadline: An absolute due time for the whole request chain. When an upstream service propagates the deadline downstream, downstream services avoid wasting work on an already-late request
Retry can only be applied safely to idempotent operations. Blindly retrying a non-idempotent operation (e.g. a payment POST) causes duplicate processing. A retry policy must therefore be designed together with the Idempotency Key/Deduplication of #28 Idempotency.
One-line answer: A method of deliberately injecting failures into the production environment to experiment with and validate system resilience — discovering weaknesses before a real outage occurs (e.g. Netflix Chaos Monkey).
Chaos Engineering is the experimental discipline of intentionally injecting failures in the production environment to build confidence that a system can withstand unexpected conditions. The idea is: "Failures will happen anyway—so let us trigger them first, in a controlled way, to find weaknesses." Distributed systems are too complex to predict behavior by reasoning alone, so we verify by experiment.
- Steady-State Hypothesis: Define the system's normal behavior in terms of measurable output (throughput, error rate, latency, etc.) and hypothesize that "this steady state will hold even during the experiment"
- Vary Real-World Events: Inject events that could plausibly happen in reality—server crashes, network latency/partition, disk exhaustion, slow dependency responses
- Run in Production: Real confidence is only verified under production traffic—carefully
- Minimize Blast Radius: Start with a small impact scope (e.g. 1% of traffic) and put a safety control in place to abort immediately
Chaos engineering began at Netflix during its migration to AWS. Chaos Monkey randomly terminated instances in production, forcing the rule "the service must stay healthy even if one instance dies." It later expanded into the Simian Army, a toolkit injecting more varied failures (Latency Monkey; Chaos Gorilla for AZ-level failure; Chaos Kong for region-level failure, etc.).
A Game Day is a drill where the team gathers at a planned time, intentionally injects failure scenarios, and practices the response. Beyond just running tools, it also validates the human response (alerting, escalation, executing runbooks). It uncovers weaknesses while building on-call response capability at the same time.
- Chaos Mesh: A Kubernetes-native chaos platform (CNCF). Declares Pod/network/IO/time faults as CRDs
- LitmusChaos: An open-source chaos framework for Kubernetes (CNCF). Composes experiments as workflows
- Gremlin: A commercial SaaS chaos-engineering platform. Strong on safety controls and a GUI
- AWS FIS (Fault Injection Service): An AWS-managed fault-injection service. Controlled fault injection into EC2/ECS/RDS, etc.
Chaos engineering surfaces weaknesses by experiment rather than by accident, removing real failures that those weaknesses would have caused. As a result, failures occur less often and MTBF (Mean Time Between Failures) increases. Rehearsing response procedures via Game Days also shortens MTTR (Mean Time To Recovery), improving overall availability.