Skip to content

Commit 34913bf

Browse files
[core] refactor: extract and centralize HTTP fetcher (#4016)
## Summary PR [#3976](#3976) introduced smart HTTP error handling for the Calendar module. This PR extracts that HTTP logic into a central `HTTPFetcher` class. Calendar is the first module to use it. Follow-up PRs would migrate Newsfeed and maybe even Weather. **Before this change:** - ❌ Each module had to implemented its own `fetch()` calls - ❌ No centralized retry logic or backoff strategies - ❌ No timeout handling for hanging requests - ❌ Error detection relied on fragile string parsing **What this PR adds:** - ✅ Unified HTTPFetcher class with intelligent retry strategies - ✅ Modern AbortController with configurable timeout (default 30s) - ✅ Proper undici Agent for self-signed certificates - ✅ Structured error objects with translation keys - ✅ Calendar module migrated as first consumer - ✅ Comprehensive unit tests with msw (Mock Service Worker) ## Architecture **Before - Decentralized HTTP handling:** ``` Calendar Module Newsfeed Module Weather Module ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ fetch() own │ │ fetch() own │ │ fetch() own │ │ retry logic │ │ basic error │ │ no retry │ │ error parse │ │ handling │ │ client-side │ └─────────────┘ └─────────────┘ └─────────────┘ │ │ │ └───────────────────────┴───────────────────────┘ ▼ External APIs ``` **After - Centralized with HTTPFetcher:** ``` ┌─────────────────────────────────────────────────────┐ │ HTTPFetcher │ │ • Unified retry strategies (401/403, 429, 5xx) │ │ • AbortController timeout (30s) │ │ • Structured errors with translation keys │ │ • undici Agent for self-signed certs │ └────────────┬──────────────┬──────────────┬──────────┘ │ │ │ ┌───────▼───────┐ ┌────▼─────┐ ┌──────▼──────┐ │ Calendar │ │ Newsfeed │ │ Weather │ │ ✅ This PR │ │ future │ │ future │ └───────────────┘ └──────────┘ └─────────────┘ │ │ │ └──────────────┴──────────────┘ ▼ External APIs ``` ## Complexity Considerations **Does HTTPFetcher add complexity?** Even if it may look more complex, it actually **reduces overall complexity**: - **Calendar already has this logic** (PR #3976) - we're extracting, not adding - **Alternative is worse:** Each module implementing own logic = 3× the code - **Better testability:** 443 lines of tests once vs. duplicating tests for each module - **Standards-based:** Retry-After is RFC 7231, not custom logic ## Future Benefits **Weather migration (future PR):** Moving Weather from client-side to server-side will enable: - **Same robust error handling** - Weather gets 429 rate-limiting, 5xx backoff for free - **Simpler architecture** - No proxy layer needed Moving the weather modules from client-side to server-side will be a big undertaking, but I think it's a good strategy. Even if we only move the calendar and newsfeed to the new HTTP fetcher and leave the weather as it is, this PR still makes sense, I think. ## Breaking Changes **None** ---- I am eager to hear your opinion on this 🙂
1 parent 23f0290 commit 34913bf

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

52 files changed

+1463
-155
lines changed

js/http_fetcher.js

Lines changed: 300 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,300 @@
1+
const { EventEmitter } = require("node:events");
2+
const { Agent } = require("undici");
3+
const Log = require("logger");
4+
const { getUserAgent } = require("#server_functions");
5+
6+
const FIFTEEN_MINUTES = 15 * 60 * 1000;
7+
const THIRTY_MINUTES = 30 * 60 * 1000;
8+
const MAX_SERVER_BACKOFF = 3;
9+
const DEFAULT_TIMEOUT = 30000; // 30 seconds
10+
11+
/**
12+
* Maps errorType to MagicMirror translation keys.
13+
* This allows HTTPFetcher to provide ready-to-use translation keys,
14+
* eliminating the need to call NodeHelper.checkFetchError().
15+
*/
16+
const ERROR_TYPE_TO_TRANSLATION = {
17+
AUTH_FAILURE: "MODULE_ERROR_UNAUTHORIZED",
18+
RATE_LIMITED: "MODULE_ERROR_RATE_LIMITED",
19+
SERVER_ERROR: "MODULE_ERROR_SERVER_ERROR",
20+
CLIENT_ERROR: "MODULE_ERROR_CLIENT_ERROR",
21+
NETWORK_ERROR: "MODULE_ERROR_NO_CONNECTION",
22+
UNKNOWN_ERROR: "MODULE_ERROR_UNSPECIFIED"
23+
};
24+
25+
/**
26+
* HTTPFetcher - Centralized HTTP fetching with intelligent error handling
27+
*
28+
* Features:
29+
* - Automatic retry strategies based on HTTP status codes
30+
* - Exponential backoff for server errors
31+
* - Retry-After header parsing for rate limiting
32+
* - Authentication support (Basic, Bearer)
33+
* - Self-signed certificate support
34+
* @augments EventEmitter
35+
* @fires HTTPFetcher#response - When fetch succeeds with ok response
36+
* @fires HTTPFetcher#error - When fetch fails or returns non-ok response
37+
* @example
38+
* const fetcher = new HTTPFetcher(url, { reloadInterval: 60000 });
39+
* fetcher.on('response', (response) => { ... });
40+
* fetcher.on('error', (errorInfo) => { ... });
41+
* fetcher.startPeriodicFetch();
42+
*/
43+
class HTTPFetcher extends EventEmitter {
44+
45+
/**
46+
* Creates a new HTTPFetcher instance
47+
* @param {string} url - The URL to fetch
48+
* @param {object} options - Configuration options
49+
* @param {number} [options.reloadInterval] - Time in ms between fetches (default: 5 min)
50+
* @param {object} [options.auth] - Authentication options
51+
* @param {string} [options.auth.method] - 'basic' or 'bearer'
52+
* @param {string} [options.auth.user] - Username for basic auth
53+
* @param {string} [options.auth.pass] - Password or token
54+
* @param {boolean} [options.selfSignedCert] - Accept self-signed certificates
55+
* @param {object} [options.headers] - Additional headers to send
56+
* @param {number} [options.maxRetries] - Max retries for 5xx errors (default: 3)
57+
* @param {number} [options.timeout] - Request timeout in ms (default: 30000)
58+
*/
59+
constructor (url, options = {}) {
60+
super();
61+
62+
this.url = url;
63+
this.reloadInterval = options.reloadInterval || 5 * 60 * 1000;
64+
this.auth = options.auth || null;
65+
this.selfSignedCert = options.selfSignedCert || false;
66+
this.customHeaders = options.headers || {};
67+
this.maxRetries = options.maxRetries || MAX_SERVER_BACKOFF;
68+
this.timeout = options.timeout || DEFAULT_TIMEOUT;
69+
70+
this.reloadTimer = null;
71+
this.serverErrorCount = 0;
72+
}
73+
74+
/**
75+
* Clears any pending reload timer
76+
*/
77+
clearTimer () {
78+
if (this.reloadTimer) {
79+
clearTimeout(this.reloadTimer);
80+
this.reloadTimer = null;
81+
}
82+
}
83+
84+
/**
85+
* Schedules the next fetch.
86+
* If no delay is provided, uses reloadInterval.
87+
* If delay is provided but very short (< 1 second), clamps to reloadInterval
88+
* to prevent hammering servers.
89+
* @param {number} [delay] - Delay in milliseconds
90+
*/
91+
scheduleNextFetch (delay) {
92+
let nextDelay = delay ?? this.reloadInterval;
93+
94+
// Only clamp if delay is unreasonably short (< 1 second)
95+
// This allows respecting Retry-After headers while preventing abuse
96+
if (nextDelay < 1000) {
97+
nextDelay = this.reloadInterval;
98+
}
99+
100+
// Don't schedule in test mode
101+
if (process.env.mmTestMode === "true") {
102+
return;
103+
}
104+
105+
this.reloadTimer = setTimeout(() => this.fetch(), nextDelay);
106+
}
107+
108+
/**
109+
* Starts periodic fetching
110+
*/
111+
startPeriodicFetch () {
112+
this.fetch();
113+
}
114+
115+
/**
116+
* Builds the options object for fetch
117+
* @returns {object} Options object containing headers (and dispatcher if needed)
118+
*/
119+
getRequestOptions () {
120+
const headers = {
121+
"User-Agent": getUserAgent(),
122+
...this.customHeaders
123+
};
124+
const options = { headers };
125+
126+
if (this.selfSignedCert) {
127+
options.dispatcher = new Agent({
128+
connect: {
129+
rejectUnauthorized: false
130+
}
131+
});
132+
}
133+
134+
if (this.auth) {
135+
if (this.auth.method === "bearer") {
136+
headers.Authorization = `Bearer ${this.auth.pass}`;
137+
} else {
138+
headers.Authorization = `Basic ${Buffer.from(`${this.auth.user}:${this.auth.pass}`).toString("base64")}`;
139+
}
140+
}
141+
142+
return options;
143+
}
144+
145+
/**
146+
* Parses the Retry-After header value
147+
* @param {string} retryAfter - The Retry-After header value
148+
* @returns {number|null} Milliseconds to wait or null if parsing failed
149+
*/
150+
#parseRetryAfter (retryAfter) {
151+
// Try parsing as seconds
152+
const seconds = Number(retryAfter);
153+
if (!Number.isNaN(seconds) && seconds >= 0) {
154+
return seconds * 1000;
155+
}
156+
157+
// Try parsing as HTTP-date
158+
const retryDate = Date.parse(retryAfter);
159+
if (!Number.isNaN(retryDate)) {
160+
return Math.max(0, retryDate - Date.now());
161+
}
162+
163+
return null;
164+
}
165+
166+
/**
167+
* Determines the retry delay for a non-ok response
168+
* @param {Response} response - The fetch Response object
169+
* @returns {{delay: number, errorInfo: object}} Computed retry delay and error info
170+
*/
171+
#getDelayForResponse (response) {
172+
const { status } = response;
173+
let delay = this.reloadInterval;
174+
let message = "";
175+
let errorType = "UNKNOWN_ERROR";
176+
177+
if (status === 401 || status === 403) {
178+
errorType = "AUTH_FAILURE";
179+
delay = Math.max(this.reloadInterval * 5, THIRTY_MINUTES);
180+
message = `Authentication failed (${status}). Waiting ${Math.round(delay / 60000)} minutes before retry.`;
181+
Log.error(`${this.url} - ${message}`);
182+
} else if (status === 429) {
183+
errorType = "RATE_LIMITED";
184+
const retryAfter = response.headers.get("retry-after");
185+
const parsed = retryAfter ? this.#parseRetryAfter(retryAfter) : null;
186+
delay = parsed !== null ? Math.max(parsed, this.reloadInterval) : Math.max(this.reloadInterval * 2, FIFTEEN_MINUTES);
187+
message = `Rate limited (429). Retrying in ${Math.round(delay / 60000)} minutes.`;
188+
Log.warn(`${this.url} - ${message}`);
189+
} else if (status >= 500) {
190+
errorType = "SERVER_ERROR";
191+
this.serverErrorCount = Math.min(this.serverErrorCount + 1, this.maxRetries);
192+
delay = this.reloadInterval * Math.pow(2, this.serverErrorCount);
193+
message = `Server error (${status}). Retry #${this.serverErrorCount} in ${Math.round(delay / 60000)} minutes.`;
194+
Log.error(`${this.url} - ${message}`);
195+
} else if (status >= 400) {
196+
errorType = "CLIENT_ERROR";
197+
delay = Math.max(this.reloadInterval * 2, FIFTEEN_MINUTES);
198+
message = `Client error (${status}). Retrying in ${Math.round(delay / 60000)} minutes.`;
199+
Log.error(`${this.url} - ${message}`);
200+
} else {
201+
message = `Unexpected HTTP status ${status}.`;
202+
Log.error(`${this.url} - ${message}`);
203+
}
204+
205+
return {
206+
delay,
207+
errorInfo: this.#createErrorInfo(message, status, errorType, delay)
208+
};
209+
}
210+
211+
/**
212+
* Creates a standardized error info object
213+
* @param {string} message - Error message
214+
* @param {number|null} status - HTTP status code or null for network errors
215+
* @param {string} errorType - Error type: AUTH_FAILURE, RATE_LIMITED, SERVER_ERROR, CLIENT_ERROR, NETWORK_ERROR
216+
* @param {number} retryAfter - Delay until next retry in ms
217+
* @param {Error} [originalError] - The original error if any
218+
* @returns {object} Error info object with translationKey for direct use
219+
*/
220+
#createErrorInfo (message, status, errorType, retryAfter, originalError = null) {
221+
return {
222+
message,
223+
status,
224+
errorType,
225+
translationKey: ERROR_TYPE_TO_TRANSLATION[errorType] || "MODULE_ERROR_UNSPECIFIED",
226+
retryAfter,
227+
retryCount: this.serverErrorCount,
228+
url: this.url,
229+
originalError
230+
};
231+
}
232+
233+
/**
234+
* Performs the HTTP fetch and emits appropriate events
235+
* @fires HTTPFetcher#response
236+
* @fires HTTPFetcher#error
237+
*/
238+
async fetch () {
239+
this.clearTimer();
240+
241+
let nextDelay = this.reloadInterval;
242+
const controller = new AbortController();
243+
const timeoutId = setTimeout(() => controller.abort(), this.timeout);
244+
245+
try {
246+
const response = await fetch(this.url, {
247+
...this.getRequestOptions(),
248+
signal: controller.signal
249+
});
250+
251+
if (!response.ok) {
252+
const { delay, errorInfo } = this.#getDelayForResponse(response);
253+
nextDelay = delay;
254+
this.emit("error", errorInfo);
255+
} else {
256+
// Reset server error count on success
257+
this.serverErrorCount = 0;
258+
259+
/**
260+
* Response event - fired when fetch succeeds
261+
* @event HTTPFetcher#response
262+
* @type {Response}
263+
*/
264+
this.emit("response", response);
265+
}
266+
} catch (error) {
267+
const isTimeout = error.name === "AbortError";
268+
const message = isTimeout ? `Request timeout after ${this.timeout}ms` : `Network error: ${error.message}`;
269+
270+
Log.error(`${this.url} - ${message}`);
271+
272+
const errorInfo = this.#createErrorInfo(
273+
message,
274+
null,
275+
"NETWORK_ERROR",
276+
this.reloadInterval,
277+
error
278+
);
279+
280+
/**
281+
* Error event - fired when fetch fails
282+
* @event HTTPFetcher#error
283+
* @type {object}
284+
* @property {string} message - Error description
285+
* @property {number|null} statusCode - HTTP status or null for network errors
286+
* @property {number} retryDelay - Ms until next retry
287+
* @property {number} retryCount - Number of consecutive server errors
288+
* @property {string} url - The URL that was fetched
289+
* @property {Error|null} originalError - The original error
290+
*/
291+
this.emit("error", errorInfo);
292+
} finally {
293+
clearTimeout(timeoutId);
294+
}
295+
296+
this.scheduleNextFetch(nextDelay);
297+
}
298+
}
299+
300+
module.exports = HTTPFetcher;

0 commit comments

Comments
 (0)