Streaming Serialization Methods for Memory-Efficient Large Subtree Processing

[freemans13]

Add Streaming Serialization Methods for Memory-Efficient Large Subtree Processing

Summary

Adds two new methods to SubtreeData that enable memory-efficient streaming serialization and deserialization of transaction data. This allows applications to process very large subtrees (1M+ transactions) without loading all transactions into memory at once.

Motivation

When processing large subtrees with millions of transactions, the existing Serialize() and serializeFromReader() methods require all transactions to be in memory simultaneously. For production workloads with 1M transaction subtrees, this results in multi-GB memory usage per subtree.

With multiple subtrees being processed concurrently, memory consumption becomes prohibitive, leading to OOM issues in production.

Changes

New Methods

WriteTransactionsToWriter(w io.Writer, startIdx, endIdx int) error

• Writes a specific range of transactions directly to a writer
• Enables streaming serialization without buffering all transactions in memory
• Transactions are written sequentially in the specified range
• Validates that transactions are non-nil before writing

ReadTransactionsFromReader(r io.Reader, startIdx, endIdx int) (int, error)

• Reads a specific range of transactions from a reader
• Enables chunked deserialization by reading only part of the data at a time
• Validates transaction hashes match expected values from subtree structure
• Returns number of transactions successfully read

Use Case

These methods enable a chunked processing pattern:

// Writing (serialization)
for chunk in chunks {
    LoadTransactionsIntoChunk()
    WriteTransactionsToWriter(writer, chunkStart, chunkEnd)
    ProcessAndReleaseChunk()  // Free memory
}

// Reading (deserialization)  
for chunk in chunks {
    ReadTransactionsFromReader(reader, chunkStart, chunkEnd)
    ProcessChunk()
    ReleaseChunk()  // Free memory
}

[github-actions]

👋 Thanks, @freemans13!

This pull request comes from a fork. For security, our CI runs in a restricted mode.
A maintainer will triage this shortly and run any additional checks as needed.

• 🏷️ Labeled: fork-pr, requires-manual-review
• 👀 We'll review and follow up here if anything else is needed.

Thanks for contributing to bsv-blockchain/go-subtree! 🚀

[mrz1836]

Woot!

[mrz1836]

Few linter issues, see CI

[mrz1836]

LGTM - however since this was a fork, tests and dep audit checks were not run. They will be run upon merging this branch.

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud