You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/docs/user-guide/dedupe.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,7 +1,7 @@
1
1
# Deduplication
2
2
3
3
With version 1.12, the crawler includes full support for deduplication ("dedupe") of crawled content by content hash,
4
-
avoiding saving the same content multuple times. When a duplicate is encountered, the crawler saves a reference to the original content, instead of the full response. The crawler supports deduplication in several ways.
4
+
avoiding saving the same content multiple times. When a duplicate is encountered, the crawler saves a reference to the original content, instead of the full response. The crawler supports deduplication in several ways.
5
5
6
6
## Automatic deduplication within a single crawl
7
7
@@ -17,7 +17,7 @@ The crawler also supports content-based deduplication by content across multiple
17
17
18
18
To enable, add `--redisDedupeUrl <redis url>` with a standard Redis format URL `redis://host:port/db`, e.g. `--redisDedupeUrl redis://my-dedupe-index:6379/0`.
19
19
20
-
Unlike the internal Redis index which is designed for the lifetime of a single crawl, this index is expected to persist accross crawls and is expected to hold all the unique hashes across many crawls.
20
+
Unlike the internal Redis index which is designed for the lifetime of a single crawl, this index is expected to persist across crawls and is expected to hold all the unique hashes across many crawls.
21
21
22
22
When enabled, the external index will store the hashes of HTTP content that has previously been archived. For each URL crawled, it'll store the unique hash and first URL and timestamp of that URL in the index. If the hash has already
23
23
been archived, it will not be saved again, instead a [`revisit` record will be created](#warc) instead.
@@ -122,7 +122,7 @@ To instead clean up the partially finished / interrupted crawls dedupe data, you
122
122
docker run -it webrecorder/browsertrix-crawler indexer --cancelCrawlId <crawl-id> --redisDedupeUrl ...
123
123
```
124
124
125
-
This operational is not required - the data from the interrupted crawl will not be used in further dedupe, it will
125
+
This operation is not required - the data from the interrupted crawl will not be used in further dedupe, it will
0 commit comments