Skip to content

Commit 456a19f

Browse files
committed
Merge branch 'feature/onyx-store-poc-baseline' into feature/onyx-store-pr-2
2 parents fa7a8d5 + 608da63 commit 456a19f

23 files changed

Lines changed: 1577 additions & 183 deletions

API-INTERNAL.md

Lines changed: 22 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -79,12 +79,18 @@ If the requested key is a collection, it will return an object with all the coll
7979
<dd><p>Remove a key from Onyx and update the subscribers</p>
8080
</dd>
8181
<dt><a href="#retryOperation">retryOperation()</a></dt>
82-
<dd><p>Handles storage operation failures based on the error type:</p>
82+
<dd><p>Handles storage operation failures based on the error class (see lib/storage/errors.ts).
83+
The connection layer (createStore) owns connection/transport recovery; this operation layer owns
84+
capacity recovery (eviction) so that a given failure is retried by exactly one layer:</p>
8385
<ul>
84-
<li>Storage capacity errors: evicts data and retries the operation</li>
85-
<li>Invalid data errors: logs an alert and throws an error</li>
86-
<li>Non-retriable errors: logs an alert and resolves without retrying</li>
87-
<li>Other errors: retries the operation</li>
86+
<li>INVALID_DATA: logs an alert and throws (the same data will always fail).</li>
87+
<li>TRANSIENT / FATAL: the connection layer already retried (transient) or exhausted its heal budget
88+
and alerted (fatal). Retrying here would only re-amplify, so we skip the write quietly.</li>
89+
<li>CAPACITY: evicts the least recently accessed evictable key and retries, under a session-level
90+
circuit breaker (see lib/StorageCircuitBreaker.ts) that halts the loop once eviction stops making
91+
progress or failures storm — the per-operation budget alone cannot stop a session-wide storm.</li>
92+
<li>UNKNOWN: the provider couldn&#39;t classify it — log the full error shape (name + message +
93+
provider) once so it&#39;s visible, then bounded retry without eviction.</li>
8894
</ul>
8995
</dd>
9096
<dt><a href="#broadcastUpdate">broadcastUpdate()</a></dt>
@@ -318,11 +324,17 @@ Remove a key from Onyx and update the subscribers
318324
<a name="retryOperation"></a>
319325

320326
## retryOperation()
321-
Handles storage operation failures based on the error type:
322-
- Storage capacity errors: evicts data and retries the operation
323-
- Invalid data errors: logs an alert and throws an error
324-
- Non-retriable errors: logs an alert and resolves without retrying
325-
- Other errors: retries the operation
327+
Handles storage operation failures based on the error class (see lib/storage/errors.ts).
328+
The connection layer (createStore) owns connection/transport recovery; this operation layer owns
329+
capacity recovery (eviction) so that a given failure is retried by exactly one layer:
330+
- INVALID_DATA: logs an alert and throws (the same data will always fail).
331+
- TRANSIENT / FATAL: the connection layer already retried (transient) or exhausted its heal budget
332+
and alerted (fatal). Retrying here would only re-amplify, so we skip the write quietly.
333+
- CAPACITY: evicts the least recently accessed evictable key and retries, under a session-level
334+
circuit breaker (see lib/StorageCircuitBreaker.ts) that halts the loop once eviction stops making
335+
progress or failures storm — the per-operation budget alone cannot stop a session-wide storm.
336+
- UNKNOWN: the provider couldn't classify it — log the full error shape (name + message +
337+
provider) once so it's visible, then bounded retry without eviction.
326338

327339
**Kind**: global function
328340
<a name="broadcastUpdate"></a>
Lines changed: 199 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,199 @@
1+
import StateMachine from '../StateMachine';
2+
import {CIRCUIT_BREAKER_TRANSITIONS} from './types';
3+
import type {CircuitBreakerOptions, CircuitBreakerState} from './types';
4+
5+
/**
6+
* Generic circuit breaker built on {@link StateMachine}.
7+
*
8+
* - **closed**: requests are allowed; failures are counted.
9+
* - **open**: requests are rejected until {@link resetTimeoutMs} elapses.
10+
* - **half-open**: the recovery-probe state. After the open timeout, the breaker admits exactly ONE
11+
* probe request: success means the dependency recovered, so the circuit closes. Failure means it's
12+
* still down, so the circuit reopens. This single-request probe prevents a "thundering herd" where
13+
* every caller fails loudly when the service hasn't recovered yet.
14+
*
15+
* Subclasses implement the failure-counting policy by overriding {@link recordFailureInClosed} (and
16+
* friends) — e.g. counting consecutive failures, or failures within a rolling time window, or any
17+
* combination of those.
18+
*
19+
* @example
20+
* class MyBreaker extends AbstractCircuitBreaker {
21+
* private failures = 0;
22+
* protected recordFailureInClosed() {
23+
* this.failures += 1;
24+
* return this.failures >= 3 ? `${this.failures} failures` : null;
25+
* }
26+
* protected recordSuccessInClosed() { this.failures = 0; }
27+
* protected resetFailureState() { this.failures = 0; }
28+
* }
29+
*
30+
* const breaker = new MyBreaker({resetTimeoutMs: 30_000});
31+
* if (breaker.isAllowed()) {
32+
* try {
33+
* doWork();
34+
* breaker.recordSuccess();
35+
* } catch {
36+
* breaker.recordFailure();
37+
* }
38+
* }
39+
*/
40+
abstract class AbstractCircuitBreaker {
41+
private machine: StateMachine<typeof CIRCUIT_BREAKER_TRANSITIONS, CircuitBreakerState>;
42+
43+
private openedAt = 0;
44+
45+
private isProbeInFlight = false;
46+
47+
private readonly resetTimeoutMs: number;
48+
49+
private readonly onTrip?: (reason: string) => void;
50+
51+
private readonly onClose?: () => void;
52+
53+
constructor(options: CircuitBreakerOptions = {}) {
54+
this.resetTimeoutMs = options.resetTimeoutMs ?? 60_000;
55+
this.onTrip = options.onTrip;
56+
this.onClose = options.onClose;
57+
this.machine = new StateMachine('closed', CIRCUIT_BREAKER_TRANSITIONS);
58+
}
59+
60+
/** Record a failure while the circuit is closed. Returns a trip reason when the threshold is exceeded. */
61+
protected abstract recordFailureInClosed(): string | null;
62+
63+
/** Update failure state after a successful request while the circuit is closed. */
64+
protected abstract recordSuccessInClosed(): void;
65+
66+
/** Clear accumulated failure state without changing circuit state. */
67+
protected abstract resetFailureState(): void;
68+
69+
/**
70+
* Whether a request may proceed.
71+
*
72+
* Returns `false` while open. In half-open, the FIRST caller is admitted as the recovery probe and
73+
* `isProbeInFlight` is latched so every subsequent caller is rejected until that probe resolves
74+
* (via {@link recordSuccess} → close, or {@link recordFailure} → reopen). That single-probe gate is
75+
* the whole point of half-open: it tests recovery with one request instead of letting a herd of
76+
* waiting callers stampede a dependency that may still be down.
77+
*/
78+
isAllowed(): boolean {
79+
const currentState = this.getCurrentState();
80+
81+
if (currentState === 'open') {
82+
return false;
83+
}
84+
85+
if (currentState === 'half-open') {
86+
if (this.isProbeInFlight) {
87+
return false;
88+
}
89+
this.isProbeInFlight = true;
90+
}
91+
92+
return true;
93+
}
94+
95+
/**
96+
* Record a failed request. May open the circuit from closed or half-open.
97+
* @returns `true` when the circuit is open after recording (the request must not proceed).
98+
*/
99+
recordFailure(): boolean {
100+
if (this.machine.state === 'open') {
101+
return true;
102+
}
103+
104+
if (this.machine.state === 'half-open') {
105+
this.trip();
106+
return true;
107+
}
108+
109+
const reason = this.recordFailureInClosed();
110+
if (reason) {
111+
this.trip(reason);
112+
return true;
113+
}
114+
115+
return false;
116+
}
117+
118+
/** Record a successful request. Closes the circuit from half-open and clears failure counts. */
119+
recordSuccess(): void {
120+
if (this.machine.state === 'half-open') {
121+
this.close();
122+
return;
123+
}
124+
125+
if (this.machine.state === 'closed') {
126+
this.recordSuccessInClosed();
127+
}
128+
}
129+
130+
/**
131+
* The current state WITHOUT advancing recovery — a pure query, safe to call without side effects.
132+
* The open→half-open transition is applied only at the admission point ({@link isAllowed}); by the
133+
* time a caller queries state after being admitted, that transition has already happened.
134+
*/
135+
peekState(): CircuitBreakerState {
136+
return this.machine.state;
137+
}
138+
139+
/**
140+
* Force the circuit back to closed from ANY state. This is a reset, not a transition, so it
141+
* deliberately bypasses the transition graph (and does not fire {@link onClose}). Use only to wipe
142+
* all state — e.g. between tests or sessions.
143+
*/
144+
protected hardReset(): void {
145+
this.machine = new StateMachine('closed', CIRCUIT_BREAKER_TRANSITIONS);
146+
this.openedAt = 0;
147+
this.isProbeInFlight = false;
148+
this.resetFailureState();
149+
}
150+
151+
private getCurrentState(): CircuitBreakerState {
152+
this.maybeRecover();
153+
return this.machine.state;
154+
}
155+
156+
private trip(reason = ''): void {
157+
if (this.machine.state === 'open') {
158+
return;
159+
}
160+
161+
this.machine = this.machine.transition('open');
162+
this.openedAt = Date.now();
163+
this.isProbeInFlight = false;
164+
this.resetFailureState();
165+
this.onTrip?.(reason);
166+
}
167+
168+
private close(): void {
169+
// close() only ever runs from half-open (see recordSuccess), and half-open → closed is the one
170+
// legal closing transition — so go through transition() to keep the illegal open → closed jump
171+
// an error rather than silently constructing a fresh closed machine.
172+
this.machine = this.machine.transition('closed');
173+
this.openedAt = 0;
174+
this.isProbeInFlight = false;
175+
this.resetFailureState();
176+
this.onClose?.();
177+
}
178+
179+
/**
180+
* Lazily advance open → half-open once the reset timeout has elapsed. This is checked on read
181+
* (via {@link getCurrentState}) rather than on a timer, so there's nothing to schedule or clean up:
182+
* the transition simply becomes visible to the next caller after the window. Entering half-open
183+
* clears `isProbeInFlight` so the next admitted request becomes the recovery probe.
184+
*/
185+
private maybeRecover(): void {
186+
if (this.machine.state !== 'open') {
187+
return;
188+
}
189+
190+
if (Date.now() - this.openedAt < this.resetTimeoutMs) {
191+
return;
192+
}
193+
194+
this.machine = this.machine.transition('half-open');
195+
this.isProbeInFlight = false;
196+
}
197+
}
198+
199+
export default AbstractCircuitBreaker;

lib/CircuitBreaker/types.ts

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
/**
2+
* States of the circuit breaker.
3+
*
4+
* - **closed**: normal operation; requests flow and failures are counted.
5+
* - **open**: tripped; requests are rejected outright so a known-bad dependency isn't hammered.
6+
* - **half-open**: a trial state entered after the open timeout — see {@link CIRCUIT_BREAKER_TRANSITIONS}.
7+
*/
8+
type CircuitBreakerState = 'closed' | 'open' | 'half-open';
9+
10+
/**
11+
* Legal state transitions. The flow is closed → open → half-open → (closed | open).
12+
*
13+
* The **half-open** state exists so the breaker can test whether the dependency has recovered WITHOUT
14+
* flipping straight back to fully closed. Going open → closed blindly would, on a dependency that is
15+
* still down, immediately re-admit the full load and re-trip — flapping between open and closed every
16+
* window. Instead, after the open timeout the breaker moves to half-open and admits a single trial
17+
* ("probe") request:
18+
* - probe succeeds → the dependency is healthy again → transition to **closed** (resume normal flow).
19+
* - probe fails → still broken → transition back to **open** for another timeout window.
20+
*
21+
* Admitting exactly one probe (rather than reopening the floodgates) is also what prevents the
22+
* "thundering herd": many callers retrying at once the instant the timeout elapses, re-overwhelming a
23+
* dependency that was just starting to recover.
24+
*/
25+
const CIRCUIT_BREAKER_TRANSITIONS = {
26+
closed: ['open'],
27+
open: ['half-open'],
28+
'half-open': ['closed', 'open'],
29+
} as const satisfies Record<CircuitBreakerState, readonly CircuitBreakerState[]>;
30+
31+
type CircuitBreakerOptions = {
32+
/** Time in milliseconds the circuit stays open before moving to half-open. */
33+
resetTimeoutMs?: number;
34+
35+
/** Called once each time the circuit opens. */
36+
onTrip?: (reason: string) => void;
37+
38+
/** Called when the circuit closes. */
39+
onClose?: () => void;
40+
};
41+
42+
export type {CircuitBreakerOptions, CircuitBreakerState};
43+
export {CIRCUIT_BREAKER_TRANSITIONS};

0 commit comments

Comments
 (0)