[UUID 2/8] UUID ingest and segment storage by xiangfu0 · Pull Request #18870 · apache/pinot

xiangfu0 · 2026-06-29T09:48:54Z

What

Ingest + segment-storage handling for UUID columns.

Changes

DataTypeTransformer normalizes UUID inputs (canonical string / java.util.UUID / byte[16]) to 16 bytes
mutable/immutable dictionaries, fixed-byte MV forward index, stats collectors, min/max, null virtual column, HashUtils
GenericRow ser/de, SegmentProcessorAvroUtils; Avro logicalType: "uuid" interop
BloomFilterCreator (segment-spi)

Enables

Offline + realtime ingest, dictionary/raw encoding, bloom filters, and UUID-backed upsert primary keys.

Depends on

#18869 (DataType.UUID, UuidUtils).

About this PR / how to review

This is part 2 of 8 splitting #18140 (first-class logical UUID type) into layered PRs, as requested there.

The split is enabled by the v1 design: DataType.UUID has stored type BYTES, so most paths handle it automatically; each PR adds explicit UUID semantics to one subsystem. Head branch lives on xiangfu0/pinot.

This PR is stacked on #18869 (branch uuid-split/01-spi-foundation). Because GitHub PRs to apache must base on master, the Files-changed tab is cumulative (it includes layers 1–2) until the PRs below it merge. Review the commit titled [UUID 2/8] UUID ingest and segment storage — that is this layer's change. Each parent merge shrinks this diff after a rebase.

Full stack (merge bottom → top)

[UUID 1/8] Add logical UUID type foundation (pinot-spi) #18869 — [UUID 1/8] logical UUID type foundation (pinot-spi)
[UUID 2/8] UUID ingest and segment storage #18870 — [UUID 2/8] UUID ingest and segment storage
[UUID 3/8] UUID result rendering (DataSchema, Arrow/JSON encoders) #18871 — [UUID 3/8] UUID result rendering (DataSchema, Arrow/JSON encoders)
[UUID 4/8] UUID server-side predicates, CAST and transforms #18872 — [UUID 4/8] UUID server-side predicates, CAST and transforms
[UUID 5/8] UUID aggregation, group-by and distinct #18873 — [UUID 5/8] UUID aggregation, group-by and distinct
[UUID 6/8] UUID multi-stage engine (planner + runtime) #18874 — [UUID 6/8] UUID multi-stage engine (planner + runtime)
[UUID 7/8] UUID scalar UDFs and partitioning #18875 — [UUID 7/8] UUID scalar UDFs and partitioning
[UUID 8/8] UUID integration tests, benchmarks and docs #18876 — [UUID 8/8] UUID integration tests, benchmarks and docs

After #18869 lands, PRs 2/4/7 (#18870, #18872, #18875) only depend on it and can be reviewed in parallel; PRs 5/6 (#18873, #18874) also need #18871; PR 8 (#18876) needs #18874.

Full feature description, v1 design contract, scope exclusions, and benchmark numbers: #18140.

codecov-commenter · 2026-06-30T10:19:12Z

Codecov Report

❌ Patch coverage is 82.94118% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.38%. Comparing base (b9e538a) to head (63d541c).

Files with missing lines	Patch %	Lines
...local/indexsegment/mutable/MutableSegmentImpl.java	60.86%	5 Missing and 4 partials ⚠️
.../segment/spi/index/creator/BloomFilterCreator.java	0.00%	6 Missing and 2 partials ⚠️
...pache/pinot/plugin/inputformat/avro/AvroUtils.java	81.25%	2 Missing and 1 partial ⚠️
...t/local/recordtransformer/DataTypeTransformer.java	89.28%	1 Missing and 2 partials ⚠️
...che/pinot/core/util/SegmentProcessorAvroUtils.java	90.90%	0 Missing and 2 partials ⚠️
.../pinot/plugin/inputformat/avro/AvroSchemaUtil.java	71.42%	0 Missing and 2 partials ⚠️
...e/impl/forward/FixedByteMVMutableForwardIndex.java	94.44%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #18870      +/-   ##
============================================
+ Coverage     57.10%   65.38%   +8.27%     
- Complexity        7     1405    +1398     
============================================
  Files          2629     3423     +794     
  Lines        156315   216098   +59783     
  Branches      25533    34213    +8680     
============================================
+ Hits          89271   141286   +52015     
- Misses        59423    63432    +4009     
- Partials       7621    11380    +3759

Flag	Coverage Δ
custom-integration1	`?`
integration	`100.00% <ø> (ø)`
integration1	`100.00% <ø> (ø)`
integration2	`?`
java-25	`65.38% <82.94%> (+8.27%)`	⬆️
temurin	`65.38% <82.94%> (+8.27%)`	⬆️
unittests	`65.37% <82.94%> (+8.26%)`	⬆️
unittests1	`57.08% <22.02%> (-0.03%)`	⬇️
unittests2	`37.59% <78.82%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot

Pull request overview

This PR adds UUID ingest- and segment-storage–level support, treating logical UUID as fixed-width 16-byte BYTES while layering UUID-specific behavior into segment-local components (forward index, dictionaries, min/max, bloom filters, Avro interop, and upsert PK normalization).

Changes:

Add UUID-aware handling in segment storage paths (MV raw forward index, mutable dictionaries, stats/min-max, null virtual column).
Add UUID normalization/validation in ingest (including upsert/dedup PK canonical-string enforcement) and UUID-aware hashing behavior for upsert PKs.
Add Avro schema interop for logicalType: "uuid" and segment-processing Avro export conversions (bytes ↔ canonical UUID string).

Reviewed changes

Copilot reviewed 36 out of 36 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/creator/BloomFilterCreator.java	Store UUID bloom entries using canonical UUID strings (rather than hex/raw forms).
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/HashUtilsTest.java	Add coverage for UUID hashing normalization and legacy-lenient UUID string handling.
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/upsert/ConcurrentMapPartitionUpsertMetadataManagerTest.java	Update mocks to provide `DataSourceMetadata.getFieldSpec()` required by updated code paths.
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/upsert/ConcurrentMapPartitionUpsertMetadataManagerForConsistentDeletesTest.java	Same as above for consistent-deletes upsert tests.
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/readers/LazyRowTest.java	Update mocks to provide `DataSourceMetadata`/`FieldSpec` for column readers.
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/columnminmaxvalue/ColumnMinMaxValueGeneratorTest.java	Add regression test ensuring raw BYTES/UUID min/max metadata direction is correct.
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/creator/BloomFilterCreatorTest.java	Add UUID bloom filter test where inputs are UUID bytes and lookups are UUID strings.
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/column/DefaultNullValueVirtualColumnProviderTest.java	Extend null-virtual-column coverage to UUID (stored as BYTES).
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/recordtransformer/DataTypeTransformerTest.java	Add tests for upsert/dedup UUID PK canonical string validation.
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/realtime/impl/dictionary/MultiValueDictionaryTest.java	Add MV BYTES dictionary test and MV raw UUID-bytes forward-index test.
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImplUuidPrimaryKeyTest.java	Add regression test for UUID PK normalization in `MutableSegmentImpl` PK construction.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/HashUtils.java	Make UUID hashing backward-compatible (lenient string parsing) while supporting non-string UUID inputs.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentColumnReader.java	Document BYTES comparator semantics also applying to canonical UUID bytes.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/BytesDictionary.java	Minor cleanup in bytes dictionary implementation.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/columnminmaxvalue/ColumnMinMaxValueGenerator.java	Fix raw BYTES/UUID min/max comparison direction (min/max inversion regression).
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexType.java	Enable fixed-width MV raw forward index for UUID (logical fixed width despite stored type BYTES).
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/column/DefaultNullValueVirtualColumnProvider.java	Treat UUID like BYTES for default-null dictionary creation.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/NoDictColumnStatisticsCollector.java	Ensure partition stats for UUID use `DataType.toString()` (canonical UUID form).
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/BytesColumnPreIndexStatsCollector.java	Same partition-stats fix for bytes/uuid pre-index stats collector.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/recordtransformer/DataTypeTransformer.java	Track UUID PK columns for upsert/dedup tables and enforce canonical lowercase UUID strings at ingest.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/forward/FixedByteMVMutableForwardIndex.java	Add MV bytes read/write support (guarded for UUID fixed-width bytes).
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/dictionary/BytesOnHeapMutableDictionary.java	Implement MV bytes indexing for on-heap bytes dictionary.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/dictionary/BytesOffHeapMutableDictionary.java	Implement MV bytes indexing for off-heap bytes dictionary.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/converter/stats/MutableNoDictColumnStatistics.java	Document comparator correctness for UUID bytes under BYTES stored type.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImpl.java	Normalize UUID PK values into `ByteArray` (and UUID-specific ordering for raw forward index sorting).
pinot-segment-local/pom.xml	Add `pinot-avro-base` as a test dependency for new Avro UUID tests.
pinot-plugins/pinot-input-format/pinot-avro-base/src/test/java/org/apache/pinot/plugin/inputformat/avro/AvroUtilsTest.java	Add schema mapping tests for Avro UUID logical type (SV + MV).
pinot-plugins/pinot-input-format/pinot-avro-base/src/test/java/org/apache/pinot/plugin/inputformat/avro/AvroSchemaUtilTest.java	Add focused tests for schema-shape mapping helpers for UUID logical types.
pinot-plugins/pinot-input-format/pinot-avro-base/src/main/java/org/apache/pinot/plugin/inputformat/avro/AvroUtils.java	Emit Avro `logicalType:uuid` for UUID fields and recognize it during schema extraction.
pinot-plugins/pinot-input-format/pinot-avro-base/src/main/java/org/apache/pinot/plugin/inputformat/avro/AvroSchemaUtil.java	Add logical-type–aware `valueOf(Schema)` and emit UUID logical type in Avro JSON schema objects.
pinot-plugins/pinot-input-format/pinot-avro-base/src/main/java/org/apache/pinot/plugin/inputformat/avro/AvroIngestionSchemaValidator.java	Fix mismatch message to report inferred Pinot type (incl. UUID logical type) rather than raw Avro type.
pinot-core/src/test/java/org/apache/pinot/core/util/SegmentProcessorAvroUtilsTest.java	Add regression tests for UUID handling when exporting GenericRow to Avro records.
pinot-core/src/test/java/org/apache/pinot/core/segment/processing/genericrow/GenericRowSerDeTest.java	Extend GenericRow ser/de tests to include MV BYTES payloads (storage shape for MV UUID).
pinot-core/src/main/java/org/apache/pinot/core/util/SegmentProcessorAvroUtils.java	Convert UUID bytes to canonical UUID strings when writing Avro with uuid logical types; emit UUID schema.
pinot-core/src/main/java/org/apache/pinot/core/segment/processing/genericrow/GenericRowSerializer.java	Add MV BYTES serialization support (needed for MV UUID round-trip in segment processing).
pinot-core/src/main/java/org/apache/pinot/core/segment/processing/genericrow/GenericRowDeserializer.java	Add MV BYTES deserialization + comparison support (mirrors serializer).

Jackie-Jiang

Please take a look at this comment: https://github.com/apache/pinot/pull/18870/changes#r3581672306

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot

Pull request overview

Copilot reviewed 38 out of 38 changed files in this pull request and generated 2 comments.

Jackie-Jiang

Mostly good. General goal is to reduce the special handling of UUID, because that usually indicates some bug in the contract

Jackie-Jiang · 2026-07-18T23:55:01Z

    SchemaBuilder.FieldAssembler<org.apache.avro.Schema> fieldAssembler = SchemaBuilder.record("record").fields();

    for (FieldSpec fieldSpec : pinotSchema.getAllFieldSpecs()) {
+      if (fieldSpec.getDataType() == DataType.UUID) {


Is this a bug in the original code? Should it switch on actual type instead of stored type?

Not a bug. This path (getAvroSchemaFromPinotSchema) intentionally switches on the stored type because it serializes Pinot's physically-stored values (BOOLEAN→int, TIMESTAMP→long, etc.). UUID is special-cased so it keeps its logical Avro form (string{logicalType:uuid}) for round-trip fidelity instead of collapsing to bytes; switching the whole method to the original type would change BOOLEAN/TIMESTAMP output too, which is out of scope here.

🤖 Addressed by Claude Code

I understand it is not introduced in this PR, but the contract should be to switch on actual data type instead of stored type. For BOOLEAN data type, it should be converted to a BOOLEAN avro type as well, instead of INT.
Also, BIG_DECIMAL is missing

Agreed the contract reads cleaner on the actual type. The catch: getAvroSchemaFromPinotSchema builds the schema for GenericRows whose values are already in stored form (BOOLEAN→Integer, TIMESTAMP→Long), so the Avro field types have to match those values — switching BOOLEAN→boolean (and adding BIG_DECIMAL) needs a coordinated value conversion, like the UUID Conversion this PR adds. Since that's a pre-existing, broader change with round-trip/compat implications beyond UUID, I'd prefer to leave this thread open and do it as a dedicated follow-up (switch on actual type + BOOLEAN/BIG_DECIMAL/TIMESTAMP logical types + value conversions) rather than fold it in here.

🤖 Addressed by Claude Code

Jackie-Jiang · 2026-07-20T23:45:27Z

    SchemaBuilder.FieldAssembler<org.apache.avro.Schema> fieldAssembler = SchemaBuilder.record("record").fields();

    for (FieldSpec fieldSpec : pinotSchema.getAllFieldSpecs()) {
+      if (fieldSpec.getDataType() == DataType.UUID) {


I understand it is not introduced in this PR, but the contract should be to switch on actual data type instead of stored type. For BOOLEAN data type, it should be converted to a BOOLEAN avro type as well, instead of INT.
Also, BIG_DECIMAL is missing

Part 2/8 of splitting apache#18140 (logical UUID type). Rebased onto master (now includes the merged apache#18869 type foundation); stacked on 92a0d98. Downstream references use the UuidKey class merged in apache#18869.

xiangfu0 force-pushed the uuid-split/02-ingest-storage branch 3 times, most recently from 1600368 to 6db6748 Compare June 30, 2026 09:14

xiangfu0 force-pushed the uuid-split/02-ingest-storage branch 3 times, most recently from ef90d01 to 6569e4a Compare July 5, 2026 19:08

xiangfu0 requested review from Jackie-Jiang and Copilot July 5, 2026 19:09

Copilot started reviewing on behalf of xiangfu0 July 5, 2026 19:10 View session

xiangfu0 added the ingestion Related to data ingestion pipeline label Jul 5, 2026

Copilot AI reviewed Jul 5, 2026

View reviewed changes

Comment thread ...al/src/main/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImpl.java Outdated

Comment thread ...segment-spi/src/main/java/org/apache/pinot/segment/spi/index/creator/BloomFilterCreator.java Outdated

xiangfu0 force-pushed the uuid-split/02-ingest-storage branch 10 times, most recently from 28af827 to 6132b99 Compare July 7, 2026 16:28

xiangfu0 force-pushed the uuid-split/02-ingest-storage branch 5 times, most recently from f8c2c0c to 1f40de6 Compare July 15, 2026 08:03

Jackie-Jiang reviewed Jul 15, 2026

View reviewed changes

xiangfu0 force-pushed the uuid-split/02-ingest-storage branch 5 times, most recently from 8a355af to 917440f Compare July 16, 2026 23:02

xiangfu0 requested review from Jackie-Jiang and Copilot July 16, 2026 23:06

Copilot started reviewing on behalf of xiangfu0 July 16, 2026 23:06 View session

Copilot AI reviewed Jul 16, 2026

xiangfu0 requested a review from Copilot July 17, 2026 00:01

Copilot started reviewing on behalf of xiangfu0 July 17, 2026 00:01 View session

Copilot AI reviewed Jul 17, 2026

View reviewed changes

Comment thread ...ocal/src/main/java/org/apache/pinot/segment/local/recordtransformer/DataTypeTransformer.java Outdated

Comment thread .../src/test/java/org/apache/pinot/segment/local/recordtransformer/DataTypeTransformerTest.java

xiangfu0 force-pushed the uuid-split/02-ingest-storage branch 7 times, most recently from d26dcdb to 5c8525e Compare July 18, 2026 09:03

Jackie-Jiang reviewed Jul 19, 2026

View reviewed changes

xiangfu0 force-pushed the uuid-split/02-ingest-storage branch 2 times, most recently from bfb965f to 16294c3 Compare July 19, 2026 00:20

Jackie-Jiang reviewed Jul 20, 2026

View reviewed changes

[UUID 2/8] UUID ingest and segment storage

63d541c

Part 2/8 of splitting apache#18140 (logical UUID type). Rebased onto master (now includes the merged apache#18869 type foundation); stacked on 92a0d98. Downstream references use the UuidKey class merged in apache#18869.

Uh oh!

Conversation

xiangfu0 commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Changes

Enables

Depends on

About this PR / how to review

Full stack (merge bottom → top)

Uh oh!

codecov-commenter commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Jackie-Jiang left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Jackie-Jiang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Jackie-Jiang Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

xiangfu0 Jul 19, 2026

Choose a reason for hiding this comment

Uh oh!

Jackie-Jiang Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

xiangfu0 Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Jackie-Jiang Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

xiangfu0 commented Jun 29, 2026 •

edited

Loading

codecov-commenter commented Jun 30, 2026 •

edited

Loading