|
1 | 1 | # Debug files upload compression findings |
2 | 2 |
|
| 3 | +## Conversation summary |
| 4 | +- `debug-files upload` does not compress whole debug files; it uploads raw bytes |
| 5 | + in chunks and only applies per-chunk transport compression (gzip or |
| 6 | + uncompressed). |
| 7 | +- There is no CLI option to compress entire debug files before upload. |
| 8 | +- The symbolic crates (v12.16.3) do not expose a whole-file DIF compression API. |
| 9 | + They only decompress compressed sections or embedded payloads when reading, |
| 10 | + and provide ZIP creation for source bundles. |
| 11 | +- `symsorter` in the symbolicator repo performs whole-file compression itself |
| 12 | + using zstd, with `-z/-zz/-zzz` mapping to zstd levels. Symbolic is used there |
| 13 | + only for parsing/iterating objects. |
| 14 | +- A symsorter-compressed file passed to `sentry-cli debug-files upload` would be |
| 15 | + skipped because format detection relies on object-file magic bytes; zstd |
| 16 | + wrappers do not match any supported DIF format. There is no whole-file |
| 17 | + decompression step in the upload path. If the format check were bypassed, the |
| 18 | + raw compressed bytes would be uploaded (still subject to per-chunk transport |
| 19 | + compression). |
| 20 | +- Extracting the symsorter compression logic into symbolic would likely require |
| 21 | + a small public helper in `symbolic-debuginfo` (or similar) that wraps zstd |
| 22 | + encoding, and then updating symsorter to call it while keeping the CLI's |
| 23 | + compression-level mapping in symsorter. |
| 24 | + |
3 | 25 | ## Question |
4 | 26 | Does `debug-files upload` ever compress the entire debug file before uploading |
5 | 27 | (not just transport-level chunk compression)? |
@@ -58,3 +80,27 @@ Does `debug-files upload` ever compress the entire debug file before uploading |
58 | 80 | - `symbolic` is used for parsing and iterating over objects (`Archive`, `Object`, |
59 | 81 | `ObjectKind`), but compression is performed in symsorter, not via symbolic |
60 | 82 | APIs. (See `symbolicator/crates/symsorter/src/app.rs` imports and usage.) |
| 83 | + |
| 84 | +## CLI compression option |
| 85 | +- `debug-files upload` does not expose a flag to compress whole debug files |
| 86 | + before upload. The upload path only applies per-chunk transport compression |
| 87 | + negotiated with the server. (See `src/commands/debug_files/upload.rs` and |
| 88 | + `src/api/mod.rs:L354-L392`.) |
| 89 | + |
| 90 | +## Symsorter-compressed file uploaded via sentry-cli |
| 91 | +- DIF discovery uses `Archive::peek` to identify object formats. If the format |
| 92 | + is `Unknown`, the file is skipped during collection. (See |
| 93 | + `src/utils/dif_upload/mod.rs:L733-L754`.) |
| 94 | +- `Archive::peek` looks for known object magic bytes (ELF, PE, Mach-O, PDB, |
| 95 | + etc.) and does not recognize zstd wrappers. (See |
| 96 | + `symbolic-debuginfo-12.16.3/src/object.rs:L127-L172`.) |
| 97 | +- There is no whole-file decompression step in the upload path, so a symsorter |
| 98 | + zstd-wrapped file would not be decompressed by sentry-cli. If the format check |
| 99 | + were bypassed, the raw compressed bytes would be uploaded. |
| 100 | + |
| 101 | +## Extracting symsorter compression into symbolic (discussion) |
| 102 | +- This would require adding a public helper (likely in `symbolic-debuginfo`) to |
| 103 | + zstd-compress raw bytes or write a compressed stream, then updating symsorter |
| 104 | + to call that helper. |
| 105 | +- The policy mapping of `-z/-zz/-zzz` to zstd levels is CLI-specific and should |
| 106 | + remain in symsorter, while symbolic would accept a raw zstd level. |
0 commit comments