BEP 52 (2/16): Merkle tree module

[meatsquirk]

Summary

Implements PR 2 from docs/BEP52-INCREMENTAL-PR-PLAN.md (Merkle Tree Module).

• Add new crates/librqbit_core/src/merkle.rs with BEP 52 merkle primitives:
  • constants: MERKLE_BLOCK_SIZE
  • typed errors: MerkleError
  • core hashing: zero_hash(), hash_block(), hash_pair()
  • construction: compute_merkle_root() returning MerkleResult { root, piece_hashes }
  • verification: verify_piece(), verify_block_with_proof()
  • helpers: padding_piece_hash(), root_from_piece_layer()
• Export the module from crates/librqbit_core/src/lib.rs (feature-gated with sha1 backends).
• Include runtime correctness guards in verification paths:
  • reject oversized piece data in verify_piece()
  • reject out-of-range block indices in verify_block_with_proof()
• Implement small-tree stack path optimization (padded leaf count <= 8) while keeping Vec path for larger trees.
• Add broad unit coverage (33 merkle tests), including proof ordering, cross-validation, trimming behavior, and invalid input cases.

BEP 52 Roadmap

This is PR 2 of 16 implementing BEP 52 (BitTorrent v2) support.
See https://gist.github.com/meatsquirk/533cae01ad9ba49834ffbb3bb3edc469 for the full stacked plan.

Test plan

• cargo +1.90.0 check --all-targets
• cargo +1.90.0 clippy --all-targets -- -D warnings -D dead_code -D unused_variables
• cargo +1.90.0 test -p librqbit-core merkle

🤖 Generated with Claude Code

Latest Update (2026-02-15)

Follow-up commit 217e3488 applies review-driven improvements to merkle.rs:

• Streaming merkle construction API: compute_merkle_root_streaming().
• Single-pass piece verification (verify_piece) with no double chunk iteration.
• Tree reduction helper reuse to avoid duplicated pair-hashing loops.
• Small-tree stack path in padding_piece_hash().
• Branchless fixed-size hash comparison (ct_eq_id32) in verification paths.
• Additional tests for streaming parity and constant-time comparison helper.

Additional validation run after this update:

• cargo check
• cargo clippy --all-targets
• cargo test -p librqbit-core merkle::tests::
• cargo test --workspace

All passed.

[ikatson]

Can we restructure the approach, so that there's no unused code?
For example, you could start with torrent metainfo / info parsing that would parse v2 fields.
Then progressively implement new things for v2 top-down (info parsing -> optionally fetching piece layers if we don't have them (magnet link case) -> validating data on disk -> downloading the missing pieces).

[meatsquirk]

I'll rework the plan to completely avoid dead code and try to vertically slice as much as possible incrementally adding functionality and minimizing risk to existing code.

[meatsquirk]

Update for PR2 after review feedback (commit 217e3488):

• Added compute_merkle_root_streaming() and kept compute_merkle_root() as a compatibility wrapper over iter().copied().
• Reduced allocation churn in tree reduction by reusing a single buffer (reduce_tree_in_place) and removed duplicated pair-hashing loops.
• Made verify_piece() single-pass over piece bytes (len().div_ceil() for block count + streaming leaves), avoiding a second chunk iteration.
• Added no-allocation root reduction path for power-of-two leaves used by verify_piece().
• Added branchless fixed-size hash comparison helper (ct_eq_id32) and switched verification equality checks to use it.
• Optimized padding_piece_hash() for small trees (<= 8 leaves) to use stack-backed path.
• Added/updated tests:
  • test_ct_eq_id32
  • test_compute_merkle_root_streaming_matches_slice_api
  • test_compute_merkle_root_streaming_without_hash_vec

Validation run after these changes:

• cargo check
• cargo clippy --all-targets
• cargo test -p librqbit-core merkle::tests::
• cargo test --workspace

All passed.

[ikatson]

As I said already, let's do other parts first, I want to see this used first before I spend time reviewing it again. It's all dead code, so let's do it only when it's actually needed (and I guess we won't need all of it, at least on the first use)