set rateLimitMaxRetries to 4 instead of infinite #1123)

default to 4 retries to avoid crawls getting stuck retrying rate limited pages indefinitely by default!

docs: updated docs to mention new default.

Author: ikreymer

docs/docs/user-guide/cli-options.md

@@ -371,10 +371,10 @@ Options:
       --rateLimitTimeout                    Time in seconds to track rate limite
                                             d count for before resetting
                                                          [number] [default: 300]
-      --rateLimitMaxRetries, --retries      If set >=0, number of times to retry
+      --rateLimitMaxRetries                 If set >=0, number of times to retry
                                              rate limited pages before marking t
                                             hem as failed. If -1, retry indefini
-                                            tely          [number] [default: -1]
+                                            tely           [number] [default: 4]
       --rateLimitInterruptCount             If >0, threshold for number of rate
                                             limited pages before crawl is consid
                                             ered rate limited and is interrupted

docs/docs/user-guide/rate-limits.md

@@ -27,10 +27,10 @@ If a site provides a `Retry-After` header, for example with a 429 response, the
 
 ## Rate Limit Retry Count
 
-By default, rate limited pages are retried indefinitely, allowing the crawl to possibly complete but at a much slower pace. If the pages are blocked due to auth requirements it may be possible to add a browser profile and restart the crawl, allowing for previously flagged pages to be captured successfully.
+By default, rate limited pages are retried 4 times, allowing the crawl to possibly complete but at a much slower pace. If the pages are blocked due to auth requirements it may be possible to add a browser profile and restart the crawl, allowing for previously flagged pages to be captured successfully.
 
-However, it may be desirable to set a total number of retries for pages flagged as rate limited, which can be done
-by setting `--rateLimitMaxRetries` to a value >= 0, where 0 implies no retries at all.
+It may be desirable to set a total number of retries for pages flagged as rate limited higher or lower, which can be done
+by setting `--rateLimitMaxRetries` to a value >= 0, where 0 implies no retries at all, while -1 means retry indefinitely.
 
 ## Interrupting Crawl on Rate Limit Threshold
 

src/util/argParser.ts

@@ -22,6 +22,7 @@ import {
   DEFAULT_RATE_LIMIT_RULES,
   RATE_LIMIT_TTL_SECS,
   RateLimitRule,
+  DEFAULT_MAX_RATE_LIMIT_RETRIES,
 } from "./constants.js";
 import { interpolateFilename } from "./storage.js";
 import { screenshotTypes } from "./screenshots.js";
@@ -790,7 +791,7 @@ class ArgParser {
           describe:
             "If set >=0, number of times to retry rate limited pages before marking them as failed. If -1, retry indefinitely",
           type: "number",
-          default: -1,
+          default: DEFAULT_MAX_RATE_LIMIT_RETRIES,
         },
 
         rateLimitInterruptCount: {

src/util/constants.ts

@@ -147,6 +147,8 @@ export const STATUS_DNS_ERROR = 899;
 // Rate Limit Constants
 export const RATE_LIMIT_TTL_SECS = 300;
 
+export const DEFAULT_MAX_RATE_LIMIT_RETRIES = 4;
+
 export type RateLimitRule = {
   regex: RegExp;
   status: number;