Support configurable merged-blocks bundle size#59
Merged
Conversation
Adds DefaultMergedBlocksBundleSize (process-wide default, like GetProtocolFirstStreamableBlock) and stream.WithMergedBlocksBundleSize for per-stream override. FileSource now errors out when a file contains blocks beyond the configured bundle size. Hub bootstrap rounding follows the configured size. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
UlysseCorbeil
requested review from
UlysseCorbeil
and removed request for
UlysseCorbeil
| out = blknum - sub | ||
| } | ||
| out = out / 100 * 100 | ||
| out = out / bundleSize * bundleSize |
There was a problem hiding this comment.
Is it "possible" to have bundleSize of 0, in config
UlysseCorbeil
approved these changes
Jul 6, 2026
FileSourceErrorOnMissingMergedBlocksFile called Shutdown() the moment the walker hit a missing file. Shutdown() fires Terminating() right away, so the consuming loop and the per-file reader goroutines abort mid-flight and discard blocks from files that were already read but not yet processed, truncating the output. Push the error through the ordered fileStream like the stop-block path does, so the consumer drains every queued block first and only then surfaces the missing-file error. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Make the merged-blocks bundle size configurable instead of hardcoded
100.Added
DefaultMergedBlocksBundleSize: package variable (default100) controlling blocks-per-merged-file assumed by readers when no explicit size is given. Set once at process startup (likeGetProtocolFirstStreamableBlock).stream.WithMergedBlocksBundleSize:streamoption to set the bundle size for a single stream (overrides the process-wide default; used by substreams tier2 which serves multiple chains at once).FileSourcefails fast with a clear error when a merged-blocks file contains a block beyond the configured bundle size.Changed
ForkableHubbootstrap rounds its lowest kept block down to the configured bundle size instead of a hardcoded100.Related
Base of the variable-bundle-size series:
feature/variable-bundle-size(pins this commit)feature/variable-bundle-sizefeature/variable-bundle-sizeMerge order: this → substreams → firehose-core → firehose-ethereum.