Skip to content

Commit b496e7e

Browse files
authored
dedupe.md - fix typos (#1105)
1 parent 35cd074 commit b496e7e

1 file changed

Lines changed: 3 additions & 3 deletions

File tree

docs/docs/user-guide/dedupe.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Deduplication
22

33
With version 1.12, the crawler includes full support for deduplication ("dedupe") of crawled content by content hash,
4-
avoiding saving the same content multuple times. When a duplicate is encountered, the crawler saves a reference to the original content, instead of the full response. The crawler supports deduplication in several ways.
4+
avoiding saving the same content multiple times. When a duplicate is encountered, the crawler saves a reference to the original content, instead of the full response. The crawler supports deduplication in several ways.
55

66
## Automatic deduplication within a single crawl
77

@@ -17,7 +17,7 @@ The crawler also supports content-based deduplication by content across multiple
1717

1818
To enable, add `--redisDedupeUrl <redis url>` with a standard Redis format URL `redis://host:port/db`, e.g. `--redisDedupeUrl redis://my-dedupe-index:6379/0`.
1919

20-
Unlike the internal Redis index which is designed for the lifetime of a single crawl, this index is expected to persist accross crawls and is expected to hold all the unique hashes across many crawls.
20+
Unlike the internal Redis index which is designed for the lifetime of a single crawl, this index is expected to persist across crawls and is expected to hold all the unique hashes across many crawls.
2121

2222
When enabled, the external index will store the hashes of HTTP content that has previously been archived. For each URL crawled, it'll store the unique hash and first URL and timestamp of that URL in the index. If the hash has already
2323
been archived, it will not be saved again, instead a [`revisit` record will be created](#warc) instead.
@@ -122,7 +122,7 @@ To instead clean up the partially finished / interrupted crawls dedupe data, you
122122
docker run -it webrecorder/browsertrix-crawler indexer --cancelCrawlId <crawl-id> --redisDedupeUrl ...
123123
```
124124

125-
This operational is not required - the data from the interrupted crawl will not be used in further dedupe, it will
125+
This operation is not required - the data from the interrupted crawl will not be used in further dedupe, it will
126126
simply free up the data on the Redis.
127127

128128
#### With concurrent crawl support

0 commit comments

Comments
 (0)