Dedupe docs (#989)

- Adds 'Deduplication' page to cover info about dedupe and indexer
entrypoint.
- Add 'indexer' section to CLI flags page.
- Put User Docs before Develop in docs menu

---------

Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>

Author: ikreymer

docs/docs/user-guide/cli-options.md

@@ -380,3 +380,22 @@ Options:
   --sshProxyKnownHostsFile  path to SSH known hosts file for SOCKS5 over SSH pro
                             xy connection                               [string]
 ```
+## indexer
+
+```
+
+Options:
+  --help            Show help                                          [boolean]
+  --version         Show version number                                [boolean]
+  --redisDedupeUrl  URL for remote redis instance to index into
+                                                             [string] [required]
+  --sourceUrl       Source WACZ or Multi WACZ or Multi WACZ JSON to index
+                                                                        [string]
+  --sourceCrawlId   If single WACZ, use this id as source id            [string]
+  --removing        If set, also remove unused crawls/hashes from index
+                                                      [boolean] [default: false]
+  --commitCrawlId   If provided, commit single uncommitted crawl to merged index
+                     and exit                                           [string]
+  --cancelCrawlId   If provided, delete data for uncommitted crawl and exit
+                                                                        [string]
+```

docs/docs/user-guide/dedupe.md

@@ -0,0 +1,139 @@
+# Deduplication
+
+With version 1.12, the crawler includes full support for deduplication ("dedupe") of crawled content by content hash,
+avoiding saving the same content multuple times. When a duplicate is encountered, the crawler saves a reference to the original content, instead of the full response. The crawler supports deduplication in several ways.
+
+## Automatic deduplication within a single crawl
+
+By default, the crawler applies the following deduplication automatically:
+
+- URL-based deduplication: If the same URL is encountered multiple times in a single crawl, the URL is automatically skipped, no revisit record is written.
+- Content-based deduplication: If a URL that points to the same *content*, but at a different URL is encountered, a `revisit` record is written,
+pointing to the first URL that has the same content. This allows for deduplication of the same content even across different URLs.
+
+## Deduplication across multiple crawls
+
+The crawler also supports content-based deduplication by content across multiple crawls, with an external Redis (or Redis-compatible database) index used to store the deduplication data.
+
+To enable, add `--redisDedupeUrl <redis url>` with a standard Redis format URL `redis://host:port/db`, e.g. `--redisDedupeUrl redis://my-dedupe-index:6379/0`.
+
+Unlike the internal Redis index which is designed for the lifetime of a single crawl, this index is expected to persist accross crawls and is expected to hold all the unique hashes across many crawls.
+
+When enabled, the external index will store the hashes of HTTP content that has previously been archived. For each URL crawled, it'll store the unique hash and first URL and timestamp of that URL in the index. If the hash has already
+been archived, it will not be saved again, instead a [`revisit` record will be created](#warc) instead.
+
+!!! tip "Using Kvrocks instead of Redis"
+  
+    While the index needs to be Redis-compatible, any Redis-compatible database can be used as well without additional changes. [Apache Kvrocks](https://kvrocks.apache.org/) is a good choice for the dedupe index database as it persists the data on disk, instead of keeping it all in memory like Redis. 
+
+When multiple crawls are running at the same time, the resources from one crawl are not yet available to the other crawls.
+This is to account for crawls that may be cancelled or fail.
+
+Once a crawl is complete, its data is fully 'committed' to the index and available to be deduplicated against by future crawls. This happens automatically when running the crawler via command line but can also be triggered [via a special indexer command](#committing-finished-crawls-to-the-index).
+
+### Example: Running crawls with deduplication
+
+Here's a quick example of running crawls with dedupe against the same index:
+
+1) Start Redis server, e.g. `redis-server -p 10000`
+
+2) Run a crawl with `--redisDedupeUrl` set, e.g. a crawl of 5 pages from `https://old.webrecorder.net/`:
+
+```sh
+docker run -it webrecorder/browsertrix-crawler crawl --redisDedupeUrl redis://localhost:10000/0 --url https://old.webrecorder.net/ --generateWACZ --collection firstCrawl --limit 5
+```
+
+3) Run a second crawl with same settings:
+
+```sh
+docker run -it webrecorder/browsertrix-crawler crawl --redisDedupeUrl redis://localhost:10000/0 --url https://old.webrecorder.net/ --generateWACZ --collection secondCrawl --limit 5
+```
+
+The second crawl should be significantly smaller than the first, as duplicate content is not written. Instead, much smaller [`revisit` records](#warc) are added for each URL that was already crawled in the first crawl.
+
+
+## Create deduplication index from existing crawls
+
+It may sometimes be useful to populate the dedupe index with existing data, e.g. one or more WACZ files of previous crawls. This can be done by reading through all the entries in the CDXJ index inside of a WACZ and populating the dedupe index based on this data.
+
+
+The crawler also includes a new dedicated entrypoint, `indexer`, to support this operation.
+
+For example, given an existing WACZ file `my-crawl.wacz`:
+
+1) Run the indexer import:
+
+```sh
+docker run -it webrecorder/browsertrix-crawler index --sourceUrl my-crawl.wacz --redisDedupeUrl redis://localhost:10000/0 
+```
+
+This will populate the dedupe index with all the content of `my-crawl.wacz`.
+
+2) It is now possible to run crawls against the deduplication index, e.g.:
+
+```sh
+docker run -it webrecorder/browsertrix-crawler crawl --redisDedupeUrl redis://localhost:10000/0 ...
+```
+
+### Removing data from deduplication index
+
+By default, the `indexer` operation is additive only, meaning that crawl data is added, but not removed.
+To remove all crawls that aren't part of `--sourceUrl` (either single WACZ or a JSON specifying multiple WACZs),
+also add the `--remove` flag. This will purge all data that is not being added.
+
+For a complete list of indexer CLI flags, see [indexer CLI flags](cli-options/#indexer).
+
+### Committing finished crawls to the index
+
+By default, the crawler will automatically commit finished crawls to the dedupe index.
+However, when running in Kubernetes (usually using the `--restartsOnError` flag), the dedupe index
+must be committed manually, since the crawl may be running in multiple instances or may be interrupted
+and restarted (or cancelled later).
+
+To commit a finished crawl, run:
+
+```sh
+docker run -it webrecorder/browsertrix-crawler indexer --commitCrawlId <crawl-id> --redisDedupeUrl ...
+```
+
+The commit process may take a long time if it was a large crawl, but generally should finish quickly.
+
+### Handling interrupted or canceled crawls
+
+If a crawler is interrupted, eg. with SIGINT, the dedupe data stored for that crawl will not yet committed.
+Since the crawl has not finished, the user may or may not want to include the data in the dedupe index.
+
+To include the partially finished crawl, run the above command with `--commitCrawlId`.
+
+To instead clean up the partially finished / interrupted crawls dedupe data, you can also run:
+
+```sh
+docker run -it webrecorder/browsertrix-crawler indexer --cancelCrawlId <crawl-id> --redisDedupeUrl ...
+```
+
+This operational is not required - the data from the interrupted crawl will not be used in further dedupe, it will
+simply free up the data on the Redis.
+
+
+## Deduplication outputs
+
+When deduplication is enabled, the crawler output is changed in the following ways (in addition to using
+less disk storage overall):
+
+### WARC
+
+The crawler will write WARC `revisit` records to indicate
+that a particular HTTP response has already been crawled. These records include a reference to the URL and timestamp
+of the original WARC `response` record, which includes the full content.
+
+HTTP headers are not deduplicated and are included in the `revisit` records.
+
+### WACZ
+
+When using deduplication, the crawler also stores dependency information between WACZ files, tracking which WACZ
+file(s) contain the WARCs with `response` records for each `revisit` record that is written.
+This allows for tracking which WACZ files are required by other WACZ files to get the full archived content.
+See the [WACZ dependency section of the developer documentation](../develop/dedupe.md#crawl-dependency-tracking-in-wacz-datapackagejson) for more details on the architecture of this dependency system.
+
+## Deduplication system architecture
+See the [developer docs for dedupe](../develop/dedupe.md) for more advanced information of the architecture of the dedupe system.

docs/gen-cli.sh

@@ -6,15 +6,24 @@ echo "# All Command-Line Options" > $out
 echo "" >> $out
 echo "The Browsertrix Crawler Docker image currently accepts the following parameters, broken down by entrypoint:" >> $out
 echo "" >> $out
+
 echo "## crawler" >> $out
 echo "" >> $out
 echo '```' >> $out
 #node $CURR/../dist/main.js --help >> $out
 docker run webrecorder/browsertrix-crawler crawl --help | tail -n +3 >> $out
 echo '```' >> $out
 echo "" >> $out
+
 echo "## create-login-profile" >> $out
 echo "" >> $out
 echo '```' >> $out
 docker run webrecorder/browsertrix-crawler create-login-profile --help | tail -n +3 >> $out
 echo '```' >> $out
+
+
+echo "## indexer" >> $out
+echo "" >> $out
+echo '```' >> $out
+docker run webrecorder/browsertrix-crawler indexer --help | tail -n +3 >> $out
+echo '```' >> $out

docs/mkdocs.yml

@@ -45,10 +45,6 @@ theme:
 
 nav:
   - index.md
-  - Develop:
-    - develop/index.md
-    - develop/docs.md
-    - develop/dedupe.md
   - User Guide:
     - user-guide/index.md
     - user-guide/outputs.md
@@ -57,10 +53,15 @@ nav:
     - user-guide/crawl-scope.md
     - user-guide/yaml-config.md
     - user-guide/browser-profiles.md
+    - user-guide/dedupe.md
     - user-guide/proxies.md
     - user-guide/behaviors.md
     - user-guide/qa.md
     - user-guide/cli-options.md
+  - Develop:
+    - develop/index.md
+    - develop/docs.md
+    - develop/dedupe.md
 
 markdown_extensions:
   - toc:

src/indexer.ts

@@ -54,7 +54,7 @@ export class CrawlIndexer {
         },
 
         removing: {
-          describe: "If set, also remove unsued crawls/hashes from index",
+          describe: "If set, also remove unused crawls/hashes from index",
           type: "boolean",
           required: false,
           default: false,