Rate Limit Docs + more flexible custom rules (#1080)

Follow-up to #1015:
- Replace --rateLimitOn200MatchText with --rateLimitOnMatch to allow
specifying either a '<regex> or '<regex>:<status_code>' rules to flag page as rate limited by regex
and optionally by status code
- Add docs for rate limit detection in user-guide/rate-limits.md

---------

Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
Co-authored-by: Misty De Méo <mistydemeo@gmail.com>

Author: 3 people

docs/docs/user-guide/cli-options.md

@@ -357,14 +357,17 @@ Options:
                                              encountered but not queued to repor
                                             ts/skippedPages.jsonl
                                                       [boolean] [default: false]
-      --rateLimitOn200MatchText             Consider page rate limited given the
-                                             following matches by status code an
-                                            d text
-                              [array] [default: ["src=\"/_Incapsula_Resource?"]]
       --rateLimitStatusCodes                Consider responses with these status
                                              codes to be treated as rate-limited
                                             /blocked responses
                                                 [array] [default: [403,429,503]]
+      --rateLimitOnMatch                    One or more rules in the format <reg
+                                            ex> or <regex>:<status code> matched
+                                             against page text to determine if a
+                                             page is rate limited. If a status i
+                                            s provided, only matches if the resp
+                                            onse has the specified status
+                          [array] [default: ["src=\"/_Incapsula_Resource?:200"]]
       --rateLimitTimeout                    Time in seconds to track rate limite
                                             d count for before resetting
                                                          [number] [default: 300]

docs/docs/user-guide/dedupe.md

@@ -88,7 +88,7 @@ By default, the `indexer` operation is additive only, meaning that crawl data is
 To remove all crawls that aren't part of `--sourceUrl` (either single WACZ or a JSON specifying multiple WACZs),
 also add the `--remove` flag. This will purge all data that is not being added.
 
-For a complete list of indexer CLI flags, see [indexer CLI flags](cli-options/#indexer).
+For a complete list of indexer CLI flags, see [indexer CLI flags](cli-options.md#indexer).
 
 ### Committing finished crawls to the index
 
@@ -181,4 +181,4 @@ See the [WACZ dependency section of the developer documentation](../develop/dedu
 If both page deduplication and [skipped page reporting](./reports.md) are enabled, pages that are skipped due to page deduplication are logged with the `duplicate` reason.
 
 ## Deduplication system architecture
-See the [developer docs for dedupe](../develop/dedupe.md) for more advanced information of the architecture of the dedupe system.
+See the [developer docs for dedupe](../develop/dedupe.md) for more advanced information of the architecture of the dedupe system.

docs/docs/user-guide/outputs.md

@@ -24,7 +24,7 @@ Additionally, the collection may include:
 - An `indexes/` directory containing merged [CDXJ](https://specs.webrecorder.net/cdxj/0.1.0/) index files for the crawl, if the `--generateCDX` or `--generateWACZ` arguments are provided. If the combined size of the CDXJ files in the `warc-cdx/` directory is over 50 KB, the resulting final CDXJ file will be gzipped.
 - A single combined gzipped [WARC](https://www.iso.org/standard/68004.html) file for the crawl, if the `--combineWARC` argument is provided.
 - A `crawls/` directory including YAML files describing the crawl state, if the `--saveState` argument is provided with a value of "always", or if the crawl is interrupted and `--saveState` is not set to "never". These files can be used to restart a crawl from its saved state.
-- A `reports/` directory containing various reports generated during the crawl. See [Reports](reports) for more info.
+- A `reports/` directory containing various reports generated during the crawl. See [Reports](reports.md) for more info.
 - 
 
 ## Profile Outputs

docs/docs/user-guide/rate-limits.md

@@ -0,0 +1,62 @@
+# Rate Limit / CAPTCHA Page Detection
+
+When crawling, it is often possible to encounter rate limits, CAPTCHAs, or auth pages that are shown instead of valid content that the crawler is intending to capture.
+These 'auth-walled' pages are often implemented for good reason, and bypassing such limits is outside of the scope of the crawler.
+
+However, the crawler can *detect* such pages and avoid archiving 'bad'/undesirable data.
+
+The crawler includes several options to help with this. 
+
+When pages are flagged as rate limited, the pages are still loaded in the browser, but the content is not archived.
+Instead, the pages may be queued to be retried at a later time or skipped altogether.
+
+## Flagging Rate Limited / Auth-walled pages
+
+The simplest mechanism for detection is by checking the status code. By default, the crawler now skips all pages that have a 403, 429, or 503 status code. This list can be customized with the `--rateLimitStatusCodes` setting, which accepts a list of status codes to always skip and retry.
+
+The crawler also includes a custom option `--rateLimitOnMatch`, which can take additional values in the form of `<regex>` or `<regex>:<status>`, e.g. `--rateLimitOnMatch <regex-1> --rateLimitOnMatch <regex-2>:<status>`. This setting allows for custom detection of CAPTCHAs and rate-limited pages based on page text content (HTML, JS, etc...), and optionally status code, so even 200 pages can be flagged as rate limited.
+
+## Rate Limit Counters
+
+The crawler also keeps track of how many pages have been flagged as limited during the duration of the crawl,
+by combination of status code and direct fetch/browser-based indicator. These stats are stored in Redis as part of the crawl stats.
+
+The crawler also keeps track of how many pages have been rate limited *within the last N seconds*, set by the `--rateLimitTimeout` and defaulting to 300 seconds (5 minutes).
+
+If a site provides a `Retry-After` header, for example with a 429 response, the header value is used instead of the global default set in `--rateLimitTimeout`.
+
+## Rate Limit Retry Count
+
+By default, rate limited pages are retried indefinitely, allowing the crawl to possibly complete but at a much slower pace. If the pages are blocked due to auth requirements it may be possible to add a browser profile and restart the crawl, allowing for previously flagged pages to be captured successfully.
+
+However, it may be desirable to set a total number of retries for pages flagged as rate limited, which can be done
+by setting `--rateLimitMaxRetries` to a value >= 0, where 0 implies no retries at all.
+
+## Interrupting Crawl on Rate Limit Threshold
+
+By default, the crawler will continue, skipping rate limited pages and retrying them indefinitely, to match existing behavior.
+
+If the `--rateLimitInterruptCount M` flag is set, the crawler will exit with a rate limit exit code (exit code 21) after M rate limited pages within the N seconds, configured via `--rateLimitTimeout`.
+
+This can allow a job runner or other system that starts and monitors the crawler container to implement an exponential backoff system if the crawler repeatedly exits due to being rate limited.
+The Browsertrix application provides this through built-in exponential backoff available in Kubernetes, where the system
+will restart the crawler in increasing intervals if continuously rate limited, up to once every 5 minutes.
+
+Additionally, if direct fetch reaches this threshold, further direct fetches will also be skipped until the timeout expires.
+
+If any page loads successfully, the rate limit counter is cleared.
+
+## Caveats and Multi-Domain Crawling
+
+It should be noted that the rate limits are applied globally and not per seed or per domain, and it may not always be desirable to interrupt a crawl containing many seeds across different domains when a rate limit is reached.
+The rate limit interrupt threshold is disabled by default for this reason.
+
+For multi-domain crawling the threshold could be set to a large number so that if multiple domains are flagged as rate limited the crawl is still interrupted, while for a single domain crawl the number may be set lower.
+It may make sense to set `--rateLimitMaxRetries` to a fixed value so that larger crawls do eventually finish with certain pages skipped as rate limited.
+
+Finally, depending on the reason for the 'auth-wall' on each domain, the crawler may or may not be able to complete the crawl on each domain.
+- For 429 errors and 503 errors, waiting and retrying may be successful.
+- For 403 errors, logging in with a browser profile may provide additional access.
+- Other sites and custom CAPTCHA pages may require additional permission to be allowed to crawl the sites.
+
+If facing repeated errors, requesting permission may be the best approach to successfully crawl a domain.

docs/mkdocs.yml

@@ -55,6 +55,7 @@ nav:
     - user-guide/browser-profiles.md
     - user-guide/dedupe.md
     - user-guide/proxies.md
+    - user-guide/rate-limits.md
     - user-guide/behaviors.md
     - user-guide/qa.md
     - user-guide/reports.md

src/util/argParser.ts

@@ -19,8 +19,9 @@ import {
   DEFAULT_MAX_RETRIES,
   BxFunctionBindings,
   DEFAULT_CRAWL_ID_TEMPLATE,
-  RATE_LIMIT_MATCH_200,
+  DEFAULT_RATE_LIMIT_RULES,
   RATE_LIMIT_TTL_SECS,
+  RateLimitRule,
   ADD_REDIRECTED_SEEDS_OPTS,
 } from "./constants.js";
 import { interpolateFilename } from "./storage.js";
@@ -61,7 +62,7 @@ export type CrawlerArgs = ReturnType<typeof parseArgs> & {
 
   warcInfo?: Record<string, string>;
 
-  rateLimitOn200MatchText: string[];
+  rateLimitCustomRules?: RateLimitRule[];
   rateLimitStatusCodes: number[];
 };
 
@@ -765,18 +766,18 @@ class ArgParser {
           default: false,
         },
 
-        rateLimitOn200MatchText: {
+        rateLimitStatusCodes: {
           describe:
-            "Consider page rate limited given the following matches by status code and text",
+            "Consider responses with these status codes to be treated as rate-limited/blocked responses",
           type: "array",
-          default: RATE_LIMIT_MATCH_200,
+          default: [403, 429, 503],
         },
 
-        rateLimitStatusCodes: {
+        rateLimitOnMatch: {
           describe:
-            "Consider responses with these status codes to be treated as rate-limited/blocked responses",
+            "One or more rules in the format <regex> or <regex>:<status code> matched against page text to determine if a page is rate limited. If a status is provided, only matches if the response has the specified status",
           type: "array",
-          default: [403, 429, 503],
+          default: DEFAULT_RATE_LIMIT_RULES,
         },
 
         rateLimitTimeout: {
@@ -975,6 +976,24 @@ class ArgParser {
       logger.info("Updating profile on successful crawl");
     }
 
+    if (argv.rateLimitOnMatch) {
+      const rules: RateLimitRule[] = [];
+      for (const rule of argv.rateLimitOnMatch) {
+        // match any status by default
+        let status = 0;
+        let regex = "";
+        if (rule.match(/:[\d]+$/)) {
+          const parts = rule.split(":");
+          regex = parts[0];
+          status = parseInt(parts[1]) ?? 0;
+        } else {
+          regex = rule;
+        }
+        rules.push({ regex: new RegExp(regex), status });
+        argv.rateLimitCustomRules = rules;
+      }
+    }
+
     return true;
   }
 }

src/util/constants.ts

@@ -143,5 +143,10 @@ export const STATUS_UNKNOWN_ERROR = 999;
 // Rate Limit Constants
 export const RATE_LIMIT_TTL_SECS = 300;
 
-// default text matches to consider rate limit on 200
-export const RATE_LIMIT_MATCH_200 = [`src="/_Incapsula_Resource?`];
+export type RateLimitRule = {
+  regex: RegExp;
+  status: number;
+};
+
+// default text matches to consider rate limit
+export const DEFAULT_RATE_LIMIT_RULES = ['src="/_Incapsula_Resource?:200'];

src/util/recorder.ts

@@ -29,6 +29,7 @@ import { ScopedSeed } from "./seeds.js";
 import EventEmitter from "events";
 import {
   DEFAULT_MAX_RETRIES,
+  RateLimitRule,
   SkippedReason,
   STATUS_IS_HTML_NO_DIRECT_FETCH,
   STATUS_UNKNOWN_ERROR,
@@ -182,7 +183,7 @@ export class Recorder extends EventEmitter {
 
   stopping = false;
 
-  rateLimitOn200MatchText: string[] = [];
+  rateLimitCustomRules: RateLimitRule[] = [];
   rateLimitStatusCodes: number[] = [];
 
   constructor({
@@ -198,7 +199,7 @@ export class Recorder extends EventEmitter {
     this.workerid = workerid;
     this.crawler = crawler;
     this.crawlState = crawler.crawlState;
-    this.rateLimitOn200MatchText = crawler.params.rateLimitOn200MatchText;
+    this.rateLimitCustomRules = crawler.params.rateLimitCustomRules ?? [];
     this.rateLimitStatusCodes = crawler.params.rateLimitStatusCodes;
 
     this.shouldSaveStorage = !!crawler.params.saveStorage;
@@ -1306,12 +1307,19 @@ export class Recorder extends EventEmitter {
 
         string = payload.toString();
 
-        if (status === 200) {
-          for (const match of this.rateLimitOn200MatchText) {
-            if (string.indexOf(match) > 0) {
-              this.markRateLimited(status, reqresp.getHeader("Retry-After"));
-              return false;
-            }
+        for (const rule of this.rateLimitCustomRules) {
+          if (
+            (rule.status === 0 || rule.status === status) &&
+            string.search(rule.regex) >= 0
+          ) {
+            const retryAfter = reqresp.getHeader("Retry-After");
+            logger.debug(
+              "Rate Limited By Custom Rule",
+              { rule: rule.regex.toString(), retryAfter },
+              "recorder",
+            );
+            this.markRateLimited(status, retryAfter);
+            return false;
           }
         }