[turbopack] Add a simple tool to inspect serialized SST files and report on their contents (#89310)

Example run
```
$ cargo run -p turbo-persistence --bin sst_inspect --  ~/projects/front/apps/vercel-site/.next/cache/turbopack/v16.2.0-canary.8-207-g8e5467e841e6e-dirty
...
Analyzing 169 SST files in /Users/lukesandberg/projects/front/apps/vercel-site/.next/cache/turbopack/v16.2.0-canary.8-207-g8e5467e841e6e-dirty

═══════════════════════════════════════════════════════════════════════════════
Family 0 (Infra):
═══════════════════════════════════════════════════════════════════════════════
  SST files: 1, Total entries: 2
  Total file size: 44 B

  Averages:
    File size: 44 B
    Keys per file: 2
    Keys per key block: 2.0

  Per-file Overhead (total): 8 B (18.2% of total file size)
    Key compression dictionaries: 0 B
      Average per file: 0 B
    Block directories: 8 B
      Average per file: 8 B

  Block Statistics:
      Index blocks: 1 blocks (uncompressed), 3 B
      Key blocks: 1 blocks (uncompressed), 25 B
      Value blocks: none

  Entry Type Histogram:
    type   9:            1 ( 50.0%) │█████████████████████████│ inline 1 bytes
    type  12:            1 ( 50.0%) │█████████████████████████│ inline 4 bytes

  Value Storage:
    Inline: 2 entries, 5 B total

═══════════════════════════════════════════════════════════════════════════════
Family 1 (TaskMeta):
═══════════════════════════════════════════════════════════════════════════════
  SST files: 36, Total entries: 8,840,774
  Total file size: 670.46 MB

  Averages:
    File size: 18.62 MB
    Keys per file: 245,577
    Keys per key block: 1358.2

  Per-file Overhead (total): 64.09 KB (0.0% of total file size)
    Key compression dictionaries: 0 B
      Average per file: 0 B
    Block directories: 64.09 KB
      Average per file: 1.78 KB

  Block Statistics:
      Index blocks: 36 blocks (uncompressed), 63.32 KB
      Key blocks: 6,509 blocks (6,478 compressed, 31 uncompressed)
          stored: 115.52 MB, actual: 134.92 MB (14% savings)
      Value blocks: 9,862 blocks (9,046 compressed, 816 uncompressed)
          stored: 554.75 MB, actual: 673.49 MB (18% savings)

  Entry Type Histogram:
    type   0:    8,840,005 (100.0%) │████████████████████████████████████████│ small value (in value block)
    type   3:          769 (  0.0%) ││ medium value

  Value Storage:
    Small (value block refs): 8,840,005 entries
    Medium (dedicated blocks): 769 entries

═══════════════════════════════════════════════════════════════════════════════
Family 2 (TaskData):
═══════════════════════════════════════════════════════════════════════════════
  SST files: 96, Total entries: 8,840,774
  Total file size: 2.60 GB

  Averages:
    File size: 27.76 MB
    Keys per file: 92,091
    Keys per key block: 1354.3

  Per-file Overhead (total): 253.12 KB (0.0% of total file size)
    Key compression dictionaries: 0 B
      Average per file: 0 B
    Block directories: 253.12 KB
      Average per file: 2.64 KB

  Block Statistics:
      Index blocks: 96 blocks (uncompressed), 63.09 KB
      Key blocks: 6,528 blocks (uncompressed), 134.84 MB
      Value blocks: 58,175 blocks (57,911 compressed, 264 uncompressed)
          stored: 2.47 GB, actual: 5.82 GB (58% savings)

  Entry Type Histogram:
    type   0:    8,825,784 ( 99.8%) │████████████████████████████████████████│ small value (in value block)
    type   3:       14,990 (  0.2%) ││ medium value

  Value Storage:
    Small (value block refs): 8,825,784 entries
    Medium (dedicated blocks): 14,990 entries

═══════════════════════════════════════════════════════════════════════════════
Family 3 (TaskCache):
═══════════════════════════════════════════════════════════════════════════════
  SST files: 36, Total entries: 8,847,914
  Total file size: 327.29 MB

  Averages:
    File size: 9.09 MB
    Keys per file: 245,775
    Keys per key block: 321.5

  Per-file Overhead (total): 2.36 MB (0.7% of total file size)
    Key compression dictionaries: 2.25 MB
      Average per file: 64.00 KB
    Block directories: 107.63 KB
      Average per file: 2.99 KB

  Block Statistics:
      Index blocks: 36 blocks (uncompressed), 268.47 KB
      Key blocks: 27,517 blocks, stored: 324.57 MB, actual: 495.61 MB (35% savings)
      Value blocks: none

  Entry Type Histogram:
    type  12:    8,847,914 (100.0%) │████████████████████████████████████████│ inline 4 bytes

  Value Storage:
    Inline: 8,847,914 entries, 33.75 MB total
```

Author: lukesandberg

turbopack/crates/turbo-persistence/Cargo.toml

@@ -42,6 +42,10 @@ rayon = { workspace = true }
 tempfile = { workspace = true }
 turbo-tasks-malloc = { workspace = true }
 
+[[bin]]
+name = "sst_inspect"
+path = "src/bin/sst_inspect.rs"
+
 [lints]
 workspace = true
 

turbopack/crates/turbo-persistence/README.md

@@ -105,7 +105,7 @@ The hashes are sorted.
 - foreach entry
   - 1 byte type
   - 3 bytes position in block after header
-- Max block size: 16 MB
+- Max block size: 16 KB
 
 A Key block contains n keys, which specify n key value pairs.
 
@@ -147,6 +147,9 @@ Depending on the `type` field entry has a different format:
 
 The entries are sorted by key hash and key.
 
+Future:
+ * Some tables have fixed sized keys with consistent values (4 byte task ids).  We could optimize key block representation in this case by skipping offset tables.
+
 #### Value Block
 
 - no header, all bytes are data referenced by other blocks
@@ -170,6 +173,7 @@ Reading start from the current sequence number and goes downwards.
       - not found -> break
     - Key Block: find key by binary search
       - found -> lookup value from value block, return
+          - read value as inline, or by using the block index in the key to find the value elsewhere in the file.
       - not found -> break
 
 ## Writing