Skip to content

Commit 05afa2c

Browse files
committed
feat(sequence): implement encode
SequenceEncoding was decode-only. Encoder validates arithmetic progression, serializes base+multiplier into SequenceMetadata proto, returns metadata-only EncodeResult (no buffers). Document the metadata-only EncodeNode pattern in CLAUDE.md for future encode stubs that follow the same shape.
1 parent 44f62ff commit 05afa2c

3 files changed

Lines changed: 148 additions & 41 deletions

File tree

CLAUDE.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -133,6 +133,19 @@ public final class FooEncoding implements Encoding {
133133

134134
Simple encodings (≤ ~80 lines total, e.g. `NullEncoding`, `BoolEncoding`) are exempt.
135135

136+
### Metadata-only encodings
137+
138+
Some encodings store all data in protobuf metadata — no buffers, no children (e.g. `SequenceEncoding`).
139+
Their `EncodeResult` uses an `EncodeNode` with `metadata` set and an empty `bufferIndices` array:
140+
141+
```java
142+
ByteBuffer metaBuf = ByteBuffer.wrap(meta.toByteArray());
143+
EncodeNode node = new EncodeNode(encodingId, metaBuf, new EncodeNode[0], new int[]{});
144+
return new EncodeResult(node, List.of(), null, null);
145+
```
146+
147+
The decoder reads back via `ctx.metadata()`, not `ctx.buffer(n)`.
148+
136149
## Testing
137150

138151
- Every feature needs unit tests covering: happy path, negative cases (invalid input, error conditions), and corner cases (empty, zero, max values, boundary conditions).

TODO.md

Lines changed: 34 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -66,46 +66,40 @@ written: /var/folders/dq/w0lpx2tj70g0cgv4ckcyth740000gn/T/junit-7248856963330236
6666
6767
## Encodings
6868
69-
### Implemented
70-
71-
| Encoding ID | Class | Decode | Encode | Encode effort | Dtypes supported |
72-
|------------------------|---------------------|--------|--------|---------------|------------------|
73-
| `vortex.primitive` | `PrimitiveEncoding` | ✅ | ✅ | — | all `PType` (I8–I64, U8–U64, F32, F64) |
74-
| `vortex.bool` | `BoolEncoding` | ✅ | ✅ | — | Bool (bit-packed) |
75-
| `vortex.dict` | `DictEncoding` | ✅ | ✅ | — | Primitive (VarBin via dict.vortex blocked by VarBinView) |
76-
| `fastlanes.delta` | `DeltaEncoding` | ✅ | ✅ | — | integer PTypes |
77-
| `fastlanes.bitpacked` | `BitpackedEncoding` | ✅ | ✅ | — | unsigned integer PTypes |
78-
| `vortex.null` | `NullEncoding` | ✅ | ✅ | — | Null |
79-
| `vortex.bytebool` | `ByteBoolEncoding` | ✅ | ✅ | — | Bool |
80-
| `vortex.zigzag` | `ZigZagEncoding` | ✅ | ✅ | — | signed integer PTypes |
81-
| `fastlanes.for` | `FrameOfReferenceEncoding` | ✅ | ✅ | — | integer PTypes |
82-
| `vortex.runend` | `RunEndEncoding` | ✅ | ✅ | — | Primitive, Utf8/Binary, Bool |
83-
| `vortex.constant` | `ConstantEncoding` | ✅ | ✅ | — | Primitive, Utf8, Binary, Bool, Null, Decimal, Extension |
84-
| `vortex.sparse` | `SparseEncoding` | ✅ | ❌ stub | — | Primitive |
85-
| `vortex.varbin` | `VarBinEncoding` | ✅ | ✅ | — | Utf8, Binary |
86-
| `vortex.sequence` | `SequenceEncoding` | ✅ | ❌ stub | — | Primitive |
87-
| `vortex.struct` | `StructEncoding` | ✅ | ❌ stub | — | Struct |
88-
| `vortex.ext` | `ExtEncoding` | ✅ | ✅ | — | Extension |
89-
| `vortex.alp` | `AlpEncoding` | ✅ | ✅ | — | F64, F32 |
90-
| `vortex.fsst` | `FsstEncoding` | ✅ | ❌ stub | — | Utf8, Binary |
91-
| `vortex.varbinview` | `VarBinViewEncoding` | ✅ | ❌ stub | — | Utf8, Binary |
92-
| `vortex.pco` | `PcoEncoding` | ❌ stub | ❌ | very hard — ANS + bin tokenization not ported | Primitive |
93-
94-
### Missing
95-
96-
| Encoding ID | Effort | Unblocks |
97-
|----------------------------|---------|-------------------------------------------------|
98-
| `vortex.chunked` | medium | `chunked.vortex` (segment-level chunked array) |
99-
| `fastlanes.rle` | medium | `rle.vortex` |
100-
| `vortex.alprd` | medium | `alprd.vortex` |
101-
| `vortex.decimal` | medium | `decimal.vortex` |
102-
| `vortex.decimal_byte_parts`| medium | `decimal_byte_parts.vortex` |
103-
| `vortex.datetimeparts` | medium | `datetimeparts.vortex` |
104-
| `vortex.list` | hard | `list.vortex` (needs list array model) |
105-
| `vortex.listview` | hard | `listview.vortex` |
106-
| `vortex.fixed_size_list` | hard | `fixed_size_list.vortex` |
107-
| `vortex.zstd` | hard | `zstd.vortex` (needs zstd native lib) |
108-
| `vortex.pco` (full) | very hard | `pco.vortex`, `tpch_*.vortex`, `clickbench_*.vortex` |
69+
### All Encodings
70+
71+
| Encoding ID | Class | Decode | Encode | Effort | Dtypes / Notes |
72+
|------------------------------|----------------------------|----------|----------|-----------|----------------|
73+
| `vortex.primitive` | `PrimitiveEncoding` | ✅ | ✅ | — | all `PType` (I8–I64, U8–U64, F32, F64) |
74+
| `vortex.bool` | `BoolEncoding` | ✅ | ✅ | — | Bool (bit-packed) |
75+
| `vortex.null` | `NullEncoding` | ✅ | ✅ | — | Null |
76+
| `vortex.bytebool` | `ByteBoolEncoding` | ✅ | ✅ | — | Bool |
77+
| `vortex.zigzag` | `ZigZagEncoding` | ✅ | ✅ | — | signed integer PTypes |
78+
| `vortex.constant` | `ConstantEncoding` | ✅ | ✅ | — | Primitive, Utf8, Binary, Bool, Null, Decimal, Extension |
79+
| `vortex.ext` | `ExtEncoding` | ✅ | ✅ | — | Extension |
80+
| `vortex.runend` | `RunEndEncoding` | ✅ | ✅ | — | Primitive, Utf8/Binary, Bool |
81+
| `vortex.varbin` | `VarBinEncoding` | ✅ | ✅ | — | Utf8, Binary |
82+
| `vortex.alp` | `AlpEncoding` | ✅ | ✅ | — | F64, F32 |
83+
| `vortex.dict` | `DictEncoding` | ✅ | ✅ | — | Primitive (VarBin via dict.vortex blocked by VarBinView) |
84+
| `fastlanes.delta` | `DeltaEncoding` | ✅ | ✅ | — | integer PTypes |
85+
| `fastlanes.bitpacked` | `BitpackedEncoding` | ✅ | ✅ | — | unsigned integer PTypes |
86+
| `fastlanes.for` | `FrameOfReferenceEncoding` | ✅ | ✅ | — | integer PTypes |
87+
| `vortex.sparse` | `SparseEncoding` | ✅ | ❌ stub | — | Primitive |
88+
| `vortex.sequence` | `SequenceEncoding` | ✅ | ✅ | — | Primitive |
89+
| `vortex.struct` | `StructEncoding` | ✅ | ❌ stub | — | Struct |
90+
| `vortex.fsst` | `FsstEncoding` | ✅ | ❌ stub | — | Utf8, Binary |
91+
| `vortex.varbinview` | `VarBinViewEncoding` | ✅ | ❌ stub | — | Utf8, Binary |
92+
| `vortex.pco` | `PcoEncoding` (stub) | ❌ | ❌ | very hard | ANS + bin tokenization not ported; unblocks `pco.vortex`, tpch/clickbench fixtures |
93+
| `vortex.chunked` | — | ❌ | ❌ | medium | unblocks `chunked.vortex` (segment-level chunked array) |
94+
| `fastlanes.rle` | — | ❌ | ❌ | medium | unblocks `rle.vortex` |
95+
| `vortex.alprd` | — | ❌ | ❌ | medium | unblocks `alprd.vortex` |
96+
| `vortex.decimal` | — | ❌ | ❌ | medium | unblocks `decimal.vortex` |
97+
| `vortex.decimal_byte_parts` | — | ❌ | ❌ | medium | unblocks `decimal_byte_parts.vortex` |
98+
| `vortex.datetimeparts` | — | ❌ | ❌ | medium | unblocks `datetimeparts.vortex` |
99+
| `vortex.list` | — | ❌ | ❌ | hard | needs list array model; unblocks `list.vortex` |
100+
| `vortex.listview` | — | ❌ | ❌ | hard | unblocks `listview.vortex` |
101+
| `vortex.fixed_size_list` | — | ❌ | ❌ | hard | unblocks `fixed_size_list.vortex` |
102+
| `vortex.zstd` | — | ❌ | ❌ | hard | needs zstd native lib; unblocks `zstd.vortex` |
109103
110104
### S3 Fixture Status (`v0.72.0/arrays/`)
111105

core/src/main/java/io/github/dfa1/vortex/encoding/SequenceEncoding.java

Lines changed: 101 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
import com.google.protobuf.InvalidProtocolBufferException;
44
import io.github.dfa1.vortex.proto.EncodingProtos;
5+
import io.github.dfa1.vortex.proto.ScalarProtos;
56
import io.github.dfa1.vortex.core.array.Array;
67
import io.github.dfa1.vortex.core.ArrayStats;
78
import io.github.dfa1.vortex.core.DType;
@@ -18,6 +19,7 @@
1819
import java.lang.foreign.MemorySegment;
1920
import java.lang.foreign.ValueLayout;
2021
import java.nio.ByteBuffer;
22+
import java.util.List;
2123

2224
/// Decoder for {@code vortex.sequence}: {@code A[i] = base + i * multiplier}.
2325
///
@@ -33,14 +35,112 @@ public EncodingId encodingId() {
3335

3436
@Override
3537
public EncodeResult encode(DType dtype, Object data) {
36-
throw new UnsupportedOperationException("encode not supported by " + encodingId());
38+
return Encoder.encode(dtype, data);
3739
}
3840

3941
@Override
4042
public Array decode(DecodeContext ctx) {
4143
return Decoder.decode(ctx);
4244
}
4345

46+
private static final class Encoder {
47+
48+
private static EncodeResult encode(DType dtype, Object data) {
49+
if (!(dtype instanceof DType.Primitive p)) {
50+
throw new VortexException(EncodingId.VORTEX_SEQUENCE, "encode only supports Primitive dtype, got " + dtype);
51+
}
52+
PType pt = p.ptype();
53+
return switch (pt) {
54+
case I8, I16, I32, I64, U8, U16, U32, U64 -> encodeInteger(pt, data);
55+
case F32 -> encodeF32((float[]) data);
56+
case F64 -> encodeF64((double[]) data);
57+
case F16 -> throw new VortexException(EncodingId.VORTEX_SEQUENCE, "F16 not supported");
58+
};
59+
}
60+
61+
private static EncodeResult encodeInteger(PType pt, Object data) {
62+
int n = intArrayLength(pt, data);
63+
long base = 0;
64+
long multiplier = 0;
65+
if (n > 0) {
66+
base = readLong(pt, data, 0);
67+
multiplier = n > 1 ? readLong(pt, data, 1) - base : 0;
68+
for (int i = 2; i < n; i++) {
69+
long expected = base + (long) i * multiplier;
70+
if (readLong(pt, data, i) != expected) {
71+
throw new VortexException(EncodingId.VORTEX_SEQUENCE, "not an arithmetic sequence at index " + i);
72+
}
73+
}
74+
}
75+
ScalarProtos.ScalarValue baseScalar = buildIntScalar(pt, base);
76+
ScalarProtos.ScalarValue mulScalar = buildIntScalar(pt, multiplier);
77+
return buildResult(baseScalar, mulScalar);
78+
}
79+
80+
private static EncodeResult encodeF32(float[] data) {
81+
float base = data.length > 0 ? data[0] : 0f;
82+
float mul = data.length > 1 ? data[1] - base : 0f;
83+
for (int i = 2; i < data.length; i++) {
84+
if (data[i] != base + (float) i * mul) {
85+
throw new VortexException(EncodingId.VORTEX_SEQUENCE, "not an arithmetic sequence at index " + i);
86+
}
87+
}
88+
ScalarProtos.ScalarValue baseScalar = ScalarProtos.ScalarValue.newBuilder().setF32Value(base).build();
89+
ScalarProtos.ScalarValue mulScalar = ScalarProtos.ScalarValue.newBuilder().setF32Value(mul).build();
90+
return buildResult(baseScalar, mulScalar);
91+
}
92+
93+
private static EncodeResult encodeF64(double[] data) {
94+
double base = data.length > 0 ? data[0] : 0.0;
95+
double mul = data.length > 1 ? data[1] - base : 0.0;
96+
for (int i = 2; i < data.length; i++) {
97+
if (data[i] != base + (double) i * mul) {
98+
throw new VortexException(EncodingId.VORTEX_SEQUENCE, "not an arithmetic sequence at index " + i);
99+
}
100+
}
101+
ScalarProtos.ScalarValue baseScalar = ScalarProtos.ScalarValue.newBuilder().setF64Value(base).build();
102+
ScalarProtos.ScalarValue mulScalar = ScalarProtos.ScalarValue.newBuilder().setF64Value(mul).build();
103+
return buildResult(baseScalar, mulScalar);
104+
}
105+
106+
private static EncodeResult buildResult(ScalarProtos.ScalarValue base, ScalarProtos.ScalarValue mul) {
107+
EncodingProtos.SequenceMetadata meta = EncodingProtos.SequenceMetadata.newBuilder()
108+
.setBase(base)
109+
.setMultiplier(mul)
110+
.build();
111+
ByteBuffer metaBuf = ByteBuffer.wrap(meta.toByteArray());
112+
EncodeNode node = new EncodeNode(EncodingId.VORTEX_SEQUENCE, metaBuf, new EncodeNode[0], new int[]{});
113+
return new EncodeResult(node, List.of(), null, null);
114+
}
115+
116+
private static ScalarProtos.ScalarValue buildIntScalar(PType pt, long value) {
117+
return switch (pt) {
118+
case U8, U16, U32, U64 -> ScalarProtos.ScalarValue.newBuilder().setUint64Value(value).build();
119+
default -> ScalarProtos.ScalarValue.newBuilder().setInt64Value(value).build();
120+
};
121+
}
122+
123+
private static int intArrayLength(PType pt, Object data) {
124+
return switch (pt) {
125+
case I8, U8 -> ((byte[]) data).length;
126+
case I16, U16 -> ((short[]) data).length;
127+
case I32, U32 -> ((int[]) data).length;
128+
case I64, U64 -> ((long[]) data).length;
129+
default -> throw new VortexException(EncodingId.VORTEX_SEQUENCE, "unsupported ptype: " + pt);
130+
};
131+
}
132+
133+
private static long readLong(PType pt, Object data, int i) {
134+
return switch (pt) {
135+
case I8, U8 -> ((byte[]) data)[i];
136+
case I16, U16 -> ((short[]) data)[i];
137+
case I32, U32 -> ((int[]) data)[i];
138+
case I64, U64 -> ((long[]) data)[i];
139+
default -> throw new VortexException(EncodingId.VORTEX_SEQUENCE, "unsupported ptype: " + pt);
140+
};
141+
}
142+
}
143+
44144
private static final class Decoder {
45145

46146
private static Array decode(DecodeContext ctx) {

0 commit comments

Comments
 (0)