Skip to content

protobuf: unrevert and fix hyperpb processor#4240

Merged
mmatczuk merged 5 commits intomainfrom
mmt/unrevert-protobuf
Apr 13, 2026
Merged

protobuf: unrevert and fix hyperpb processor#4240
mmatczuk merged 5 commits intomainfrom
mmt/unrevert-protobuf

Conversation

@mmatczuk
Copy link
Copy Markdown
Contributor

@mmatczuk mmatczuk commented Apr 10, 2026

  • Revert "protobuf: remove hyperpb"
  • protobuf: fix leak in hyperpb parser
  • protobuf: guard against zero RecompileInterval in hyperpb decoder

if err := msg.Unmarshal(buf, hyperpb.WithRecordProfile(state.profile, p.opts.Rate)); err != nil {
return err
}
if state.profile != nil && p.seen.Add(1)%p.opts.RecompileInterval == 0 {
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Integer division by zero — if RecompileInterval is 0 (the zero value of int64), the modulo operation p.seen.Add(1) % p.opts.RecompileInterval will panic at runtime.

Since ProfilingOptions is an exported struct with no validation, and NewHyperPbDecoder always creates a non-nil profile (line 30), the state.profile != nil guard won't protect against this — the first call to WithDecoded will panic if a zero-value ProfilingOptions{} is passed.

Add a guard (e.g., p.opts.RecompileInterval > 0 &&) before the modulo, or validate RecompileInterval in NewHyperPbDecoder.

@claude
Copy link
Copy Markdown

claude Bot commented Apr 10, 2026

Commits
LGTM

Review
Reverts the hyperpb removal and fixes a memory leak by calling shared.Free() before returning Shared objects to the pool. The concurrent state-swap logic for profile-guided recompilation is sound.

  1. Division by zero in hyperPbParser.WithDecodeddecode_hyperpb.go:79: p.seen.Add(1) % p.opts.RecompileInterval panics when RecompileInterval is 0 (the zero value of int64). Since ProfilingOptions is exported with no validation and NewHyperPbDecoder always creates a non-nil profile, passing a zero-value struct causes an immediate panic on the first message.

@mmatczuk mmatczuk changed the title unrevert hyperpb protobuf processor protobuf: unrevert and fix hyperpb processor Apr 10, 2026
Comment on lines 88 to 93
cachedDecoder = common.NewHyperPbDecoder(msgDesc, common.ProfilingOptions{
Rate: 0.01,
RecompileInterval: 100_000,
})
}
return cachedDecoder
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The literal values 0.01 and 100_000 for profiling rate and recompile interval are repeated in three locations across two files (serde_protobuf.go, processor_protobuf.go x2) without named constants. Per the project's Magic Numbers code style rule:

Name all numeric constants. Every literal number in logic must have a clear meaning through a named constant or variable.

Define these as named constants (or a package-level default ProfilingOptions) in common and reference them from all three call sites. This also reduces the risk of the values diverging across locations.

@claude
Copy link
Copy Markdown

claude Bot commented Apr 10, 2026

Commits
LGTM

Review
This PR re-introduces hyperpb as a profile-guided optimized protobuf decoder with platform-aware fallback, fixes a pool-related memory leak, and guards against division by zero on RecompileInterval. The concurrency design (atomic state swap, CAS-gated background recompile) is sound.

  1. Magic numbers for profiling options (serde_protobuf.go#L88-L93): The literal values 0.01 and 100_000 are repeated across three call sites in two files without named constants, violating the project's Magic Numbers rule. Define a shared default in common and reference it from all callers.

@claude
Copy link
Copy Markdown

claude Bot commented Apr 10, 2026

Commits
LGTM

Review
This PR re-introduces the hyperpb decoder with proper fixes: a memory leak fix via shared.Free() before pool return, a guard against zero RecompileInterval to prevent modulo-by-zero panics, and extraction of DefaultProfilingOptions to eliminate magic literals. The concurrency design is solid — atomic pointer swaps for state, CAS-guarded recompilation, and pool-per-state to ensure old pools drain cleanly with the old state. Platform fallback via complementary build tags (arm64 || amd64 vs \!(arm64 || amd64)) is correct. All callers of the changed NewDynamicPbDecoder signature have been updated.

LGTM

rockwotj and others added 4 commits April 10, 2026 13:24
@mmatczuk mmatczuk force-pushed the mmt/unrevert-protobuf branch from b2733d9 to 95c6253 Compare April 10, 2026 11:24
Comment on lines +76 to +77
if err := msg.Unmarshal(buf, hyperpb.WithRecordProfile(state.profile, p.opts.Rate)); err != nil {
return err
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing error wrapping: The unmarshal error is returned bare, but the existing dynamicPbParser.WithDecoded wraps it with message-type context:

return fmt.Errorf("unmarshalling protobuf message: '%v': %w", p.msgType.Descriptor().FullName(), err)

Since ProtobufDecoder implementations are interchangeable, error messages should be consistent. Wrap the error here with equivalent context (message type full name) so callers get the same diagnostic information regardless of which decoder is active. This also aligns with the project error handling pattern requiring fmt.Errorf wrapping with %w.

@claude
Copy link
Copy Markdown

claude Bot commented Apr 10, 2026

Commits
LGTM

Review
This PR reintroduces the hyperpb decoder with a profile-guided recompilation strategy, a fallback to dynamicpb on unsupported platforms, and benchmarks comparing both decoders. The concurrency design (atomic state swaps, pool-per-state for GC-friendly lifecycle) is solid.

  1. Missing error wrapping in hyperPbParser.WithDecoded (decode_hyperpb.go#L76-L77): The unmarshal error is returned bare (return err), but the existing dynamicPbParser wraps with message-type context. Since the two implementations are interchangeable behind the ProtobufDecoder interface, error messages should be consistent. This also violates the project error handling pattern requiring fmt.Errorf wrapping with %w.

@claude
Copy link
Copy Markdown

claude Bot commented Apr 10, 2026

Commits
LGTM

Review
This PR re-introduces the hyperpb decoder with a fix for the pool/GC leak and adds profile-guided optimization (PGO) support with configurable profiling options. The concurrency design is sound: atomic state swaps with CAS for exclusive recompilation, pool-per-state isolation for safe GC, and a guard against zero RecompileInterval. The fallback on unsupported platforms gracefully degrades to dynamicpb.

LGTM

@mmatczuk mmatczuk merged commit 692d35a into main Apr 13, 2026
7 checks passed
@mmatczuk mmatczuk deleted the mmt/unrevert-protobuf branch April 13, 2026 13:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants