Page Level Dedupe support: (#1018)

- add --dedupePagesMinDepth to enable page-level dedupe at certain depth
or greater
- add 'duplicate' as another skip reason, log skip reason when page is
skipped due to dedupe
- when pageDedupe is enabled, set pageLimit to 0 and allow queueing
pages beyond expected limit, in case pages are skipped
- add queuePageLimit and check limit on each new page at queue pop time,
allows skipping already deduped pages and incrementally crawling new
pages
- when limit reached, queued pages are drained and marked as excluded /
logged to skippedPages list
- tests: test page dedupe / incremental crawling: new pages are archived
on subsequent crawls, previous pages skipped with 'duplicate' reason
- docs: add Page Deduplication on dedupe page
- docs: add Reports page (reports.md), document skipped pages /
--reportSkipped report
- fixes #1017

---------

Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>

Author: ikreymer

docs/docs/user-guide/cli-options.md

@@ -194,6 +194,10 @@ Options:
       --redisDedupeUrl                      If set, url for remote redis server
                                             to store state. Otherwise, using loc
                                             al redis instance           [string]
+      --dedupePagesMinDepth                 If set >= 0, minimum depth at which
+                                            duplicate pages can be skipped. -1 m
+                                            eans never skip duplicate pages
+                                                          [number] [default: -1]
       --saveState                           If the crawl state should be seriali
                                             zed to the crawls/ directory. Defaul
                                             ts to 'partial', only saved when cra

docs/docs/user-guide/dedupe.md

@@ -115,6 +115,28 @@ This operational is not required - the data from the interrupted crawl will not
 simply free up the data on the Redis.
 
 
+## Page Deduplication
+
+The crawler also has the option to skip loading of entire pages if the page HTML is a duplicate.
+
+In such cases, after the `revisit` record for the HTML page is written, the crawler aborts loading the page in the browser and moves on to the next page.
+
+This allows saving not only storage but crawling time, as duplicate pages are quickly skipped by the browser.
+
+Note that there is a trade-off involved, as any page resources (images, stylesheets) or links are also skipped and are not crawled, even if they may have changed.
+
+To enable this feature, set the `--dedupePagesMinDepth` to a value of 0 or greater.
+Setting to a value of 0 means that even the seed page will be skipped if it has not changed, and no additional pages will be crawled.
+
+It is generally recommended to set `--dedupePagesMinDepth` to a value of at least 1 when using this feature. Setting to a value of 1 will ensure that the seed page is always fully crawled and that its links will be added to the crawl queue. Pages one link away from the seed and at greater depths will be loaded and then skipped if they are unchanged.
+
+
+### Incremental Crawling
+
+When duplicate pages are skipped they do not count towards the page limit, and additional pages can be crawled up to the page limit. This approach can be used to crawl a site that hasn't changed incrementally, a few pages at a time.
+
+For example, if a site has 100 pages, a home page and 99 other static pages that don't change, it can be fully crawled after 11 crawls with the setting `--dedupePagesMinDepth 10 --pageLimit 10`, which will crawl the home page and up to 9 additional pages each time the crawler is run.
+
 ## Deduplication outputs
 
 When deduplication is enabled, the crawler output is changed in the following ways (in addition to using
@@ -135,5 +157,9 @@ file(s) contain the WARCs with `response` records for each `revisit` record that
 This allows for tracking which WACZ files are required by other WACZ files to get the full archived content.
 See the [WACZ dependency section of the developer documentation](../develop/dedupe.md#crawl-dependency-tracking-in-wacz-datapackagejson) for more details on the architecture of this dependency system.
 
+### Reporting
+
+If both page deduplication and [skipped page reporting](./reports.md) are enabled, pages that are skipped due to page deduplication are logged with the `duplicate` reason.
+
 ## Deduplication system architecture
 See the [developer docs for dedupe](../develop/dedupe.md) for more advanced information of the architecture of the dedupe system.

docs/docs/user-guide/outputs.md

@@ -9,16 +9,23 @@ Browsertrix Crawler crawl outputs are organized into collections, which can be f
 Each collection is a directory which contains at minimum:
 
 - `archive/`: A directory containing gzipped [WARC](https://www.iso.org/standard/68004.html) files containing the web traffic recorded during crawling.
+- `downloads/`: A directory containing any resources the crawler downloaded to run the crawl, such as browser profiles or behaviors.
 - `logs/`: A directory containing one or more crawler log files in [JSON-Lines](https://jsonlines.org/) format.
 - `pages/`: A directory containing one or more "Page" files in [JSON-Lines](https://jsonlines.org/) format. At minimum, this directory will contain a `pages.jsonl` file with information about the seed URLs provided to the crawler. If additional pages were discovered and in scope during crawling, information about those non-seed pages is written to `extraPages.jsonl`. For more information about the contents of Page files, see the [WACZ specification](https://specs.webrecorder.net/wacz/1.1.1/#pages-jsonl).
-- `warc-cdx/`: A directory containing one or more [CDXJ](https://specs.webrecorder.net/cdxj/0.1.0/) index files created while recording traffic to WARC files. These index files are 
+- `profile/`: A directory containing the current browser profile, either new or loaded from a saved profile.
+- `crawlIds/`: A directory containing a single `crawlIds.txt` file which lists the identifiers of all crawls included in this collection.
+- `warc-cdx/`: A directory containing one or more [CDXJ](https://specs.webrecorder.net/cdxj/0.1.0/) index files created while recording traffic to WARC files. These index files contain entries for every WARC record written and are updated
+at the same time as the WARCs.
+
 
 Additionally, the collection may include:
 
 - A WACZ file named after the collection, if the `--generateWACZ` argument is provided.
 - An `indexes/` directory containing merged [CDXJ](https://specs.webrecorder.net/cdxj/0.1.0/) index files for the crawl, if the `--generateCDX` or `--generateWACZ` arguments are provided. If the combined size of the CDXJ files in the `warc-cdx/` directory is over 50 KB, the resulting final CDXJ file will be gzipped.
 - A single combined gzipped [WARC](https://www.iso.org/standard/68004.html) file for the crawl, if the `--combineWARC` argument is provided.
 - A `crawls/` directory including YAML files describing the crawl state, if the `--saveState` argument is provided with a value of "always", or if the crawl is interrupted and `--saveState` is not set to "never". These files can be used to restart a crawl from its saved state.
+- A `reports/` directory containing various reports generated during the crawl. See [Reports](reports) for more info.
+- 
 
 ## Profile Outputs
 

docs/docs/user-guide/reports.md

@@ -0,0 +1,27 @@
+# Reports
+
+Browsertrix has the option to generate optional reports with each crawl. The following reports are currently available.
+Browsertrix has the option to generate optional reports with each crawl. All reports are in the JSONL format, with one JSON entry per line. The following reports are currently available.
+
+## Skipped Pages Report
+
+Written to `reports/skippedPages.jsonl` and enabled with `--reportSkipped`, this report is in the same format as the `pages/pages.jsonl` file, but lists pages that were either never loaded or where page loading was immediately aborted and no content was archived from that page.
+
+Each line in the report contains the following:
+
+- `url`: Page URL
+- `ts`: The ISO Date of the time the page was encountered
+- `seedUrl`: The seed URL that this page was discovered from
+- `depth`: The depth of the page if it were to be crawled
+- `seed`: true|false if the page is a seed
+- `reason`: Reason for skipping this page
+
+### Skip Reasons
+
+The `reason` may be one of the following:
+
+- `outOfScope`: Page URL out of scope according to scoping rules
+- `pageLimit`: The limit `--pageLimit` was reached before the page could be crawled
+- `robotsTxt`: The page URL has been excluded via robots.txt rules
+- `redirectToExcluded`: A special case of `outOfScope` where the page URL itself is in scope but loading it resulted in a HTTP redirect to a page that was not in scope, so page loading was aborted
+- `duplicate`: The page content is a duplicate and loading was aborted (see [Page Deduplication](./dedupe.md#page-deduplication) for more information)

docs/mkdocs.yml

@@ -57,6 +57,7 @@ nav:
     - user-guide/proxies.md
     - user-guide/behaviors.md
     - user-guide/qa.md
+    - user-guide/reports.md
     - user-guide/cli-options.md
   - Develop:
     - develop/index.md

src/crawler.ts

@@ -128,6 +128,7 @@ export class Crawler {
 
   limitHit = false;
   pageLimit: number;
+  queuePageLimit: number;
 
   saveStateFiles: string[] = [];
   lastSaveTime: number;
@@ -260,6 +261,13 @@ export class Crawler {
         : this.params.maxPageLimit;
     }
 
+    // if using page dedupe, set queuePageLimit to 0
+    // so queue can be unlimited, as unknown how many pages may be skipped
+    // and additional pages needed from the queue
+    // otherwise, queuePageLimit == pagesLimit
+    this.queuePageLimit =
+      this.params.dedupePagesMinDepth >= 0 ? 0 : this.pageLimit;
+
     this.saveStateFiles = [];
     this.lastSaveTime = 0;
 
@@ -613,7 +621,7 @@ export class Crawler {
 
     await this.loadCrawlState();
 
-    await this.crawlState.trimToLimit(this.pageLimit);
+    await this.crawlState.trimToLimit(this.queuePageLimit);
   }
 
   extraChromeArgs() {
@@ -1385,12 +1393,9 @@ self.__bx_behaviors.selectMainBehavior();
       if (pageSkipped) {
         await this.crawlState.markExcluded(url);
 
-        this.writeSkippedPage(
-          url,
-          data.seedId,
-          depth,
-          SkippedReason.RedirectToExcluded,
-        );
+        if (data.pageSkipReason) {
+          this.writeSkippedPage(url, data.seedId, depth, data.pageSkipReason);
+        }
         this.limitHit = false;
       } else {
         const retry = await this.crawlState.markFailed(url, noRetries);
@@ -2746,7 +2751,7 @@ self.__bx_behaviors.selectMainBehavior();
 
     const result = await this.crawlState.addToQueue(
       { url, seedId, depth, extraHops, ts, pageid },
-      this.pageLimit,
+      this.queuePageLimit,
     );
 
     switch (result) {
@@ -2978,7 +2983,7 @@ self.__bx_behaviors.selectMainBehavior();
       headers,
       fromDate,
       toDate,
-      limit: this.pageLimit,
+      limit: this.queuePageLimit,
     });
 
     let power = 1;

src/util/argParser.ts

@@ -457,12 +457,12 @@ class ArgParser {
           type: "string",
         },
 
-        // minPageDedupeDepth: {
-        //   describe:
-        //     "If set >= 0, minimum depth at which duplicate pages can be skipped. -1 means never skip duplicate pages",
-        //   type: "number",
-        //   default: -1,
-        // },
+        dedupePagesMinDepth: {
+          describe:
+            "If set >= 0, minimum depth at which duplicate pages can be skipped. -1 means never skip duplicate pages",
+          type: "number",
+          default: -1,
+        },
 
         saveState: {
           describe:

src/util/constants.ts

@@ -123,4 +123,5 @@ export enum SkippedReason {
   PageLimit = "pageLimit",
   RobotsTxt = "robotsTxt",
   RedirectToExcluded = "redirectToExcluded",
+  Duplicate = "duplicate",
 }

src/util/recorder.ts

@@ -27,8 +27,13 @@ import { Crawler } from "../crawler.js";
 import { getProxyDispatcher } from "./proxy.js";
 import { ScopedSeed } from "./seeds.js";
 import EventEmitter from "events";
-import { DEFAULT_MAX_RETRIES, WARC_REFERS_TO_CONTAINER } from "./constants.js";
+import {
+  DEFAULT_MAX_RETRIES,
+  SkippedReason,
+  WARC_REFERS_TO_CONTAINER,
+} from "./constants.js";
 import { Readable } from "stream";
+import { createHash } from "crypto";
 
 const MAX_BROWSER_DEFAULT_FETCH_SIZE = 5_000_000;
 const MAX_TEXT_REWRITE_SIZE = 25_000_000;
@@ -137,6 +142,7 @@ export class Recorder extends EventEmitter {
   mainFrameId: string | null = null;
   skipRangeUrls!: Map<string, number>;
   skipPageInfo = false;
+  state: PageState | null = null;
 
   swTargetId?: string | null;
   swFrameIds = new Set<string>();
@@ -160,7 +166,7 @@ export class Recorder extends EventEmitter {
 
   pageSeed?: ScopedSeed;
   pageSeedDepth = 0;
-  //minPageDedupeDepth = -1;
+  dedupePagesMinDepth = -1;
 
   frameIdToExecId: Map<string, number> | null;
 
@@ -184,7 +190,7 @@ export class Recorder extends EventEmitter {
 
     this.shouldSaveStorage = !!crawler.params.saveStorage;
 
-    //this.minPageDedupeDepth = crawler.params.minPageDedupeDepth;
+    this.dedupePagesMinDepth = crawler.params.dedupePagesMinDepth;
 
     this.writer = writer;
 
@@ -863,32 +869,33 @@ export class Recorder extends EventEmitter {
 
     const rewritten = await this.rewriteResponse(reqresp, mimeType);
 
-    // ** WIP: Experimental page-level dedupe **
-    // Will abort page loading in case of duplicate
-    // TODO: Write revisit record, track page as a duplicate in page list
-    // if (
-    //   url === this.pageUrl &&
-    //   reqresp.payload &&
-    //   this.minPageDedupeDepth >= 0 &&
-    //   this.pageSeedDepth >= this.minPageDedupeDepth
-    // ) {
-    //   const hash =
-    //     "sha256:" + createHash("sha256").update(reqresp.payload).digest("hex");
-    //   const res = await this.crawlState.getHashDupe(hash);
-    //   if (res) {
-    //     const { index, crawlId } = res;
-    //     const errorReason = "BlockedByResponse";
-    //     await cdp.send("Fetch.failRequest", {
-    //       requestId,
-    //       errorReason,
-    //     });
-    //     await this.crawlState.addDupeCrawlDependency(crawlId, index);
-    //     // await this.crawlState.addConservedSizeStat(
-    //     //   size - reqresp.payload.length,
-    //     // );
-    //     return true;
-    //   }
-    // }
+    // If page is at dedupePagesMinDepth or higher and HTML is a duplicate,
+    // write a revisit record, track pages as a duplicate, and abort the page
+    if (
+      url === this.pageUrl &&
+      reqresp.payload &&
+      this.dedupePagesMinDepth >= 0 &&
+      this.pageSeedDepth >= this.dedupePagesMinDepth
+    ) {
+      const hash =
+        "sha256:" + createHash("sha256").update(reqresp.payload).digest("hex");
+      const res = await this.crawlState.getHashDupe(hash);
+      if (res) {
+        await cdp.send("Fetch.failRequest", {
+          requestId,
+          errorReason: "BlockedByResponse",
+        });
+        await this.serializeToWARC(reqresp, undefined, false, true, hash);
+        this.skipPageInfo = true;
+        this.state!.pageSkipReason = SkippedReason.Duplicate;
+        logger.debug(
+          "Skipped loading duplicate page",
+          { pageUrl: url, depth: this.pageSeedDepth, ...this.logDetails },
+          "pageStatus",
+        );
+        return true;
+      }
+    }
 
     // not rewritten, and not streaming, return false to continue
     if (!rewritten && !streamingConsume) {
@@ -978,6 +985,7 @@ export class Recorder extends EventEmitter {
           loc = new URL(loc, url).href;
 
           if (this.pageSeed && this.pageSeed.isExcluded(loc)) {
+            this.state!.pageSkipReason = SkippedReason.RedirectToExcluded;
             logger.warn(
               "Skipping page that redirects to excluded URL",
               { newUrl: loc, origUrl: this.pageUrl },
@@ -994,7 +1002,15 @@ export class Recorder extends EventEmitter {
     }
   }
 
-  startPage({ pageid, url }: { pageid: string; url: string }) {
+  startPage({
+    pageid,
+    url,
+    state,
+  }: {
+    pageid: string;
+    url: string;
+    state: PageState;
+  }) {
     this.pageid = pageid;
     this.pageUrl = url;
     this.finalPageUrl = this.pageUrl;
@@ -1012,6 +1028,7 @@ export class Recorder extends EventEmitter {
     this.skipRangeUrls = new Map<string, number>();
     this.skipPageInfo = false;
     this.pageFinished = false;
+    this.state = state;
     this.pageInfo = {
       pageid,
       urls: {},
@@ -1653,10 +1670,14 @@ export class Recorder extends EventEmitter {
     reqresp: RequestResponseInfo,
     iter?: AsyncIterable<Uint8Array>,
     canRetry = false,
+    skipPageInfo = false,
+    matchHash?: string,
   ): Promise<SerializeRes> {
     // always include in pageinfo record if going to serialize to WARC
     // even if serialization does not happen, indicates this URL was on the page
-    this.addPageRecord(reqresp);
+    if (!skipPageInfo) {
+      this.addPageRecord(reqresp);
+    }
 
     const { pageid, gzip } = this;
     const { url, status, requestId, method, payload } = reqresp;
@@ -1758,6 +1779,13 @@ export class Recorder extends EventEmitter {
     if (!isEmpty && url) {
       const res = await this.crawlState.getHashDupe(hash);
 
+      // if only writing revisit, ensure it's a revisit and hash
+      // matches expected value, otherwise just return
+      // should always match, but just in case!
+      if (matchHash && (matchHash !== hash || !res)) {
+        return SerializeRes.Skipped;
+      }
+
       if (res) {
         const { origUrl, origDate, crawlId, index, size } = res;
         origRecSize = size;
@@ -1828,8 +1856,6 @@ export class Recorder extends EventEmitter {
       addStatsCallback,
     );
 
-    this.addPageRecord(reqresp);
-
     return SerializeRes.Success;
   }
 }

src/util/state.ts

@@ -12,6 +12,7 @@ import {
   DUPE_ALL_COUNTS,
   DUPE_UNCOMMITTED,
   CrawlStatus,
+  SkippedReason,
 } from "./constants.js";
 import { ScopedSeed } from "./seeds.js";
 import { Frame } from "puppeteer-core";
@@ -98,6 +99,7 @@ export class PageState {
 
   skipBehaviors = false;
   pageSkipped = false;
+  pageSkipReason: SkippedReason | null = null;
   noRetries = false;
 
   isDirectFetched = false;