rpk/serde: replace hamba/avro and linkedin/goavro with twmb/avro#29983
Merged
r-vasquez merged 1 commit intoredpanda-data:devfrom Apr 9, 2026
Merged
rpk/serde: replace hamba/avro and linkedin/goavro with twmb/avro#29983r-vasquez merged 1 commit intoredpanda-data:devfrom
r-vasquez merged 1 commit intoredpanda-data:devfrom
Conversation
Contributor
Author
|
/dt |
r-vasquez
force-pushed
the
avro-migration
branch
from
cbde9ad to
e958923
Compare
r-vasquez
requested review from
Copilot and
graham-rp
and removed request for
a team
Contributor
There was a problem hiding this comment.
Pull request overview
Replaces the existing split Avro implementation in rpk (hamba/avro for reference resolution + linkedin/goavro for encode/decode) with a single implementation based on twmb/avro, updating both the Go module dependencies and Bazel deps accordingly.
Changes:
- Switch Avro encode/decode to use
github.com/twmb/avroschemas (Encode/Decode) and JSON marshal/unmarshal. - Implement explicit schema reference resolution via
avro.SchemaCache(instead of relying on hamba/avro’s implicit global cache). - Update Go/Bazel dependency graphs to remove hamba/avro + goavro and add twmb/avro.
Reviewed changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
src/go/rpk/pkg/serde/serde.go |
Wires NewSerde to use the new compiled *avro.Schema for Avro serdes. |
src/go/rpk/pkg/serde/BUILD |
Replaces Bazel deps for Avro from hamba/goavro to twmb/avro. |
src/go/rpk/pkg/serde/avro.go |
Reimplements Avro encode/decode and reference parsing using twmb/avro and SchemaCache. |
src/go/rpk/go.mod |
Drops hamba/goavro + replace directive, adds github.com/twmb/avro. |
src/go/rpk/go.sum |
Updates sums consistent with the dependency switch (including removing old Avro libs). |
MODULE.bazel |
Updates bzlmod use_repo entries to remove old Avro repos and add com_github_twmb_avro. |
graham-rp
reviewed
Apr 9, 2026
Contributor
graham-rp
left a comment
graham-rp
left a comment
There was a problem hiding this comment.
Agreed with copilot about a test for union encoding, but looks good otherwise
c-julin
previously approved these changes
Apr 9, 2026
Replace the two separate avro libraries (hamba/avro for schema reference
resolution, linkedin/goavro for encoding/decoding) with the single
twmb/avro library, which handles both.
This changes union serialization output: decoded union values are now
written as bare JSON values rather than the Avro-JSON {"type": value}
form used by linkedin/goavro. The encoder accepts both formats as input.
Encoding and decoding now use twmb/avro's schema-aware DecodeJSON and
EncodeJSON methods, which preserve full int64 precision for Avro long
fields (json.Unmarshal loses precision beyond 2^53).
Schema reference resolution now uses an explicit SchemaCache rather than
relying on hamba/avro's implicit global cache. This also removes the
replace directive for the redpanda-data/go-avro fork.
(cherry picked from commit 59dcf96)
r-vasquez
force-pushed
the
avro-migration
branch
from
e958923 to
dd8b425
Compare
Collaborator
Retry command for Build#82959please wait until all jobs are finished before running the slash command |
Collaborator
Contributor
Author
|
/ci-repeat 1 |
graham-rp
approved these changes
Apr 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Continuation of #29845 ...
Replace the two separate avro libraries (hamba/avro for schema reference resolution, linkedin/goavro for encoding/decoding) with the single twmb/avro library, which handles both.
This changes union serialization behavior: linkedin/goavro used Avro-JSON encoding where unions are serialized as {"type": value} objects. twmb/avro marshals union values as the bare type directly. Similarly for deserialization, we will no longer read the old {"type": value} union form but instead expect the value directly.
Schema reference resolution now uses an explicit SchemaCache rather than relying on hamba/avro's implicit global cache. This also removes the replace directive for the redpanda-data/go-avro fork.
(cherry picked from commit 59dcf96)
Backports Required
Release Notes