You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+6-10Lines changed: 6 additions & 10 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -110,8 +110,8 @@ Now that we've looked at the uncompressed versions of these files to understand
110
110
The [warcio](https://github.com/webrecorder/warcio) Python library lets us read and write WARC files programmatically.
111
111
```mermaid
112
112
flowchart LR
113
-
user["user (r/w)"]--warcio (r) -->warc
114
-
user--warcio (w) -->warc
113
+
user["userprocess (r/w)"]--warcio (w) -->warc
114
+
warc --warcio (r)--> user
115
115
warc@{shape: cyl}
116
116
```
117
117
Let's use it to iterate over our WARC, WET, and WAT files and print out the record types we looked at before. First, look at the code in `warcio-iterator.py`:
@@ -177,11 +177,13 @@ The output has three sections, one each for the WARC, WET, and WAT. For each one
177
177
178
178
## Task 3: Index the WARC, WET, and WAT
179
179
180
-
The example WARC files we've been using are tiny and easy to work with. The real WARC files are around a gigabyte in size and contain about 30,000 webpages each. What's more, we have around 24 million of these files! To read all of them, we could iterate, but what if we wanted random access so we could read just one particular record? We do that with an index.
180
+
The example WARC files we've been using a\e tiny and easy to work with. The real WARC files are around a gigabyte in size and contain about 30,000 webpages each. What's more, we have around 24 million of these files! To read all of them, we could iterate, but what if we wanted random access so we could read just one particular record? We do that with an index.
@@ -335,12 +337,6 @@ Make sure you compress WARCs the right way!
335
337
## Task 6: Use cdx_toolkit to query the full CDX index and download those captures from AWS S3
336
338
337
339
Some of our users only want to download a small subset of the crawl. They want to run queries against an index, either the CDX index we just talked about, or in the columnar index, which we'll talk about later.
338
-
```mermaid
339
-
flowchart LR
340
-
user --cdx_toolkit--> cdxi
341
-
cdxi@{shape: cyl}
342
-
```
343
-
344
340
345
341
The [cdx_toolkit](https://github.com/cocrawler/cdx_toolkit) is a set of tools for working with CDX indices of web crawls and archives. It knows how to query the CDX index across all of our crawls and also can create WARCs of just the records you want. We will fetch the same record from Wikipedia that we've been using for the whirlwind tour.
0 commit comments