[UUID 2/8] UUID ingest and segment storage

[xiangfu0]

What

Ingest + segment-storage handling for UUID columns.

Changes

• DataTypeTransformer normalizes UUID inputs (canonical string / java.util.UUID / byte[16]) to 16 bytes
• mutable/immutable dictionaries, fixed-byte MV forward index, stats collectors, min/max, null virtual column, HashUtils
• GenericRow ser/de, SegmentProcessorAvroUtils; Avro logicalType: "uuid" interop
• BloomFilterCreator (segment-spi)

Enables

Offline + realtime ingest, dictionary/raw encoding, bloom filters, and UUID-backed upsert primary keys.

Depends on

#18869 (DataType.UUID, UuidUtils).

About this PR / how to review

This is part 2 of 8 splitting #18140 (first-class logical UUID type) into layered PRs, as requested there.

The split is enabled by the v1 design: DataType.UUID has stored type BYTES, so most paths handle it automatically; each PR adds explicit UUID semantics to one subsystem. Head branch lives on xiangfu0/pinot.

This PR is stacked on #18869 (branch uuid-split/01-spi-foundation). Because GitHub PRs to apache must base on master, the Files-changed tab is cumulative (it includes layers 1–2) until the PRs below it merge. Review the commit titled [UUID 2/8] UUID ingest and segment storage — that is this layer's change. Each parent merge shrinks this diff after a rebase.

Full stack (merge bottom → top)

1. [UUID 1/8] Add logical UUID type foundation (pinot-spi) #18869 — [UUID 1/8] logical UUID type foundation (pinot-spi)
2. [UUID 2/8] UUID ingest and segment storage #18870 — [UUID 2/8] UUID ingest and segment storage
3. [UUID 3/8] UUID result rendering (DataSchema, Arrow/JSON encoders) #18871 — [UUID 3/8] UUID result rendering (DataSchema, Arrow/JSON encoders)
4. [UUID 4/8] UUID server-side predicates, CAST and transforms #18872 — [UUID 4/8] UUID server-side predicates, CAST and transforms
5. [UUID 5/8] UUID aggregation, group-by and distinct #18873 — [UUID 5/8] UUID aggregation, group-by and distinct
6. [UUID 6/8] UUID multi-stage engine (planner + runtime) #18874 — [UUID 6/8] UUID multi-stage engine (planner + runtime)
7. [UUID 7/8] UUID scalar UDFs and partitioning #18875 — [UUID 7/8] UUID scalar UDFs and partitioning
8. [UUID 8/8] UUID integration tests, benchmarks and docs #18876 — [UUID 8/8] UUID integration tests, benchmarks and docs

After #18869 lands, PRs 2/4/7 (#18870, #18872, #18875) only depend on it and can be reviewed in parallel; PRs 5/6 (#18873, #18874) also need #18871; PR 8 (#18876) needs #18874.

Full feature description, v1 design contract, scope exclusions, and benchmark numbers: #18140.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 82.94118% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.40%. Comparing base (fae8080) to head (97b8541).

Files with missing lines	Patch %	Lines
...local/indexsegment/mutable/MutableSegmentImpl.java	60.86%	5 Missing and 4 partials ⚠️
.../segment/spi/index/creator/BloomFilterCreator.java	0.00%	6 Missing and 2 partials ⚠️
...pache/pinot/plugin/inputformat/avro/AvroUtils.java	81.25%	2 Missing and 1 partial ⚠️
...t/local/recordtransformer/DataTypeTransformer.java	89.28%	1 Missing and 2 partials ⚠️
...che/pinot/core/util/SegmentProcessorAvroUtils.java	90.90%	0 Missing and 2 partials ⚠️
.../pinot/plugin/inputformat/avro/AvroSchemaUtil.java	71.42%	0 Missing and 2 partials ⚠️
...e/impl/forward/FixedByteMVMutableForwardIndex.java	94.44%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #18870      +/-   ##
============================================
+ Coverage     65.37%   65.40%   +0.03%     
  Complexity     1405     1405              
============================================
  Files          3423     3423              
  Lines        215972   216102     +130     
  Branches      34186    34213      +27     
============================================
+ Hits         141199   141350     +151     
+ Misses        63399    63373      -26     
- Partials      11374    11379       +5     

Flag	Coverage Δ
custom-integration1	100.00% <ø> (ø)
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-21	65.40% <82.94%> (+0.03%)	⬆️
temurin	65.40% <82.94%> (+0.03%)	⬆️
unittests	65.40% <82.94%> (+0.03%)	⬆️
unittests1	57.08% <22.02%> (-0.01%)	⬇️
unittests2	37.63% <78.82%> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

This PR adds UUID ingest- and segment-storage–level support, treating logical UUID as fixed-width 16-byte BYTES while layering UUID-specific behavior into segment-local components (forward index, dictionaries, min/max, bloom filters, Avro interop, and upsert PK normalization).

Changes:

• Add UUID-aware handling in segment storage paths (MV raw forward index, mutable dictionaries, stats/min-max, null virtual column).
• Add UUID normalization/validation in ingest (including upsert/dedup PK canonical-string enforcement) and UUID-aware hashing behavior for upsert PKs.
• Add Avro schema interop for logicalType: "uuid" and segment-processing Avro export conversions (bytes ↔ canonical UUID string).

Reviewed changes

Copilot reviewed 36 out of 36 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/creator/BloomFilterCreator.java	Store UUID bloom entries using canonical UUID strings (rather than hex/raw forms).
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/HashUtilsTest.java	Add coverage for UUID hashing normalization and legacy-lenient UUID string handling.
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/upsert/ConcurrentMapPartitionUpsertMetadataManagerTest.java	Update mocks to provide DataSourceMetadata.getFieldSpec() required by updated code paths.
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/upsert/ConcurrentMapPartitionUpsertMetadataManagerForConsistentDeletesTest.java	Same as above for consistent-deletes upsert tests.
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/readers/LazyRowTest.java	Update mocks to provide DataSourceMetadata/FieldSpec for column readers.
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/columnminmaxvalue/ColumnMinMaxValueGeneratorTest.java	Add regression test ensuring raw BYTES/UUID min/max metadata direction is correct.
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/creator/BloomFilterCreatorTest.java	Add UUID bloom filter test where inputs are UUID bytes and lookups are UUID strings.
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/column/DefaultNullValueVirtualColumnProviderTest.java	Extend null-virtual-column coverage to UUID (stored as BYTES).
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/recordtransformer/DataTypeTransformerTest.java	Add tests for upsert/dedup UUID PK canonical string validation.
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/realtime/impl/dictionary/MultiValueDictionaryTest.java	Add MV BYTES dictionary test and MV raw UUID-bytes forward-index test.
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImplUuidPrimaryKeyTest.java	Add regression test for UUID PK normalization in MutableSegmentImpl PK construction.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/HashUtils.java	Make UUID hashing backward-compatible (lenient string parsing) while supporting non-string UUID inputs.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentColumnReader.java	Document BYTES comparator semantics also applying to canonical UUID bytes.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/BytesDictionary.java	Minor cleanup in bytes dictionary implementation.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/columnminmaxvalue/ColumnMinMaxValueGenerator.java	Fix raw BYTES/UUID min/max comparison direction (min/max inversion regression).
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexType.java	Enable fixed-width MV raw forward index for UUID (logical fixed width despite stored type BYTES).
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/column/DefaultNullValueVirtualColumnProvider.java	Treat UUID like BYTES for default-null dictionary creation.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/NoDictColumnStatisticsCollector.java	Ensure partition stats for UUID use DataType.toString() (canonical UUID form).
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/BytesColumnPreIndexStatsCollector.java	Same partition-stats fix for bytes/uuid pre-index stats collector.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/recordtransformer/DataTypeTransformer.java	Track UUID PK columns for upsert/dedup tables and enforce canonical lowercase UUID strings at ingest.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/forward/FixedByteMVMutableForwardIndex.java	Add MV bytes read/write support (guarded for UUID fixed-width bytes).
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/dictionary/BytesOnHeapMutableDictionary.java	Implement MV bytes indexing for on-heap bytes dictionary.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/dictionary/BytesOffHeapMutableDictionary.java	Implement MV bytes indexing for off-heap bytes dictionary.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/converter/stats/MutableNoDictColumnStatistics.java	Document comparator correctness for UUID bytes under BYTES stored type.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImpl.java	Normalize UUID PK values into ByteArray (and UUID-specific ordering for raw forward index sorting).
pinot-segment-local/pom.xml	Add pinot-avro-base as a test dependency for new Avro UUID tests.
pinot-plugins/pinot-input-format/pinot-avro-base/src/test/java/org/apache/pinot/plugin/inputformat/avro/AvroUtilsTest.java	Add schema mapping tests for Avro UUID logical type (SV + MV).
pinot-plugins/pinot-input-format/pinot-avro-base/src/test/java/org/apache/pinot/plugin/inputformat/avro/AvroSchemaUtilTest.java	Add focused tests for schema-shape mapping helpers for UUID logical types.
pinot-plugins/pinot-input-format/pinot-avro-base/src/main/java/org/apache/pinot/plugin/inputformat/avro/AvroUtils.java	Emit Avro logicalType:uuid for UUID fields and recognize it during schema extraction.
pinot-plugins/pinot-input-format/pinot-avro-base/src/main/java/org/apache/pinot/plugin/inputformat/avro/AvroSchemaUtil.java	Add logical-type–aware valueOf(Schema) and emit UUID logical type in Avro JSON schema objects.
pinot-plugins/pinot-input-format/pinot-avro-base/src/main/java/org/apache/pinot/plugin/inputformat/avro/AvroIngestionSchemaValidator.java	Fix mismatch message to report inferred Pinot type (incl. UUID logical type) rather than raw Avro type.
pinot-core/src/test/java/org/apache/pinot/core/util/SegmentProcessorAvroUtilsTest.java	Add regression tests for UUID handling when exporting GenericRow to Avro records.
pinot-core/src/test/java/org/apache/pinot/core/segment/processing/genericrow/GenericRowSerDeTest.java	Extend GenericRow ser/de tests to include MV BYTES payloads (storage shape for MV UUID).
pinot-core/src/main/java/org/apache/pinot/core/util/SegmentProcessorAvroUtils.java	Convert UUID bytes to canonical UUID strings when writing Avro with uuid logical types; emit UUID schema.
pinot-core/src/main/java/org/apache/pinot/core/segment/processing/genericrow/GenericRowSerializer.java	Add MV BYTES serialization support (needed for MV UUID round-trip in segment processing).
pinot-core/src/main/java/org/apache/pinot/core/segment/processing/genericrow/GenericRowDeserializer.java	Add MV BYTES deserialization + comparison support (mirrors serializer).

[Jackie-Jiang]

Please take a look at this comment: https://github.com/apache/pinot/pull/18870/changes#r3581672306

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Pull request overview

Copilot reviewed 38 out of 38 changed files in this pull request and generated 2 comments.

[Jackie-Jiang]

Mostly good. General goal is to reduce the special handling of UUID, because that usually indicates some bug in the contract