Skip to content

Commit 5b1151e

Browse files
dfa1claude
dfa1
and
claude
committed
build: commit generated fbs/proto sources, simplify CI
Generated FlatBuffers (fbs/) and Protobuf (proto/) Java sources are now committed under core/src/main/java so normal builds need no external tools (no flatc, no protoc). Moved flatc/protoc invocations into a `regenerate-sources` Maven profile: ./mvnw generate-sources -pl core -P regenerate-sources An antrun step strips ValidateVersion() from regenerated fbs files. Reason: flatbuffers-java on Maven Central (25.2.10) lags the flatc CLI by months (brew installs 25.12.19 as of 2026-06). Generated code references Constants.FLATBUFFERSXXX() that does not exist in the published jar. The binary wire format is stable across releases so the strip is safe. Full explanation in core/pom.xml. Removed flatc/protoc install steps from CI workflow. Updated CLAUDE.md and README with regeneration instructions. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 83216a0 commit 5b1151e

43 files changed

Lines changed: 33549 additions & 361 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/ci.yml‎

Lines changed: 0 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -31,15 +31,5 @@ jobs:
3131
restore-keys: |
3232
${{ runner.os }}-maven-
3333
34-
- name: Install protobuf compiler
35-
run: sudo apt-get install -y protobuf-compiler
36-
37-
- name: Install flatc 25.12.19
38-
run: |
39-
curl -fsSL -o flatc.zip https://github.com/google/flatbuffers/releases/download/v25.12.19/Linux.flatc.binary.g++-13.zip
40-
unzip -j flatc.zip flatc
41-
sudo install -m 755 flatc /usr/local/bin/flatc
42-
flatc --version
43-
4434
- name: Build and test
4535
run: ./mvnw verify

‎CLAUDE.md‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,18 @@ Keep commits small and `main` always green.
1515

1616
## Commands
1717

18-
Build prerequisites: `brew install flatbuffers protobuf` (flatc + protoc must be on PATH).
1918
Never use `mvn install` or `./mvwn install`.
2019

20+
Generated sources (`fbs`/`proto` → Java) are committed under `core/src/main/java`.
21+
Normal builds need no external tools.
22+
To regenerate after editing `.fbs` or `.proto` schemas:
23+
```bash
24+
brew install flatbuffers protobuf
25+
./mvnw generate-sources -pl core -P regenerate-sources
26+
# then commit the updated files
27+
```
28+
Any `flatc` version works — the profile strips the version guard automatically.
29+
2130
```bash
2231
# Build all modules
2332
./mvnw verify

‎README.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -188,7 +188,10 @@ java -jar cli/target/vortex.jar import data/trades.csv
188188
## Requirements
189189
190190
- Java 25+
191-
- `flatc` and `protoc` on `PATH` (build-time only: `brew install flatbuffers protobuf`)
191+
192+
`flatc` and `protoc` are **not** required for normal builds — generated sources are committed.
193+
To regenerate after editing `.fbs`/`.proto` schemas: `brew install flatbuffers protobuf && ./mvnw generate-sources -pl core -P regenerate-sources`
194+
(any `flatc` version works — the profile strips the version guard automatically).
192195
193196
Java 25 is the minimum because the FFM API (`MemorySegment`, `Arena`) was finalized as a
194197
standard API in JDK 22 (JEP 454) — it was preview/incubator in JDK 19–21 and required

‎core/pom.xml‎

Lines changed: 113 additions & 112 deletions
Original file line numberDiff line numberDiff line change
@@ -46,120 +46,121 @@
4646
</dependency>
4747
</dependencies>
4848

49-
<build>
50-
<plugins>
49+
<!--
50+
Generated sources (src/main/java/…/fbs and …/proto) are committed to the repo.
51+
Normal builds need no external tools.
5152
52-
<!-- 1. Generate Java from FlatBuffer schemas -->
53-
<plugin>
54-
<groupId>org.codehaus.mojo</groupId>
55-
<artifactId>exec-maven-plugin</artifactId>
56-
<executions>
57-
<execution>
58-
<id>flatc-generate</id>
59-
<phase>generate-sources</phase>
60-
<goals>
61-
<goal>exec</goal>
62-
</goals>
63-
<configuration>
64-
<executable>flatc</executable>
65-
<workingDirectory>${project.basedir}/src/main/flatbuffers</workingDirectory>
66-
<arguments>
67-
<argument>--java</argument>
68-
<argument>-I</argument>
69-
<argument>.</argument>
70-
<argument>-o</argument>
71-
<argument>${project.build.directory}/generated-sources/flatbuffers</argument>
72-
<argument>vortex-file/footer.fbs</argument>
73-
<argument>vortex-layout/layout.fbs</argument>
74-
<argument>vortex-array/array.fbs</argument>
75-
<argument>vortex-dtype/dtype.fbs</argument>
76-
</arguments>
77-
</configuration>
78-
</execution>
79-
<!-- 2. Generate Java from Protobuf schemas -->
80-
<execution>
81-
<id>protoc-mkdir</id>
82-
<phase>generate-sources</phase>
83-
<goals>
84-
<goal>exec</goal>
85-
</goals>
86-
<configuration>
87-
<executable>mkdir</executable>
88-
<arguments>
89-
<argument>-p</argument>
90-
<argument>${project.build.directory}/generated-sources/protobuf</argument>
91-
</arguments>
92-
</configuration>
93-
</execution>
94-
<execution>
95-
<id>protoc-generate</id>
96-
<phase>generate-sources</phase>
97-
<goals>
98-
<goal>exec</goal>
99-
</goals>
100-
<configuration>
101-
<executable>protoc</executable>
102-
<arguments>
103-
<argument>--java_out=${project.build.directory}/generated-sources/protobuf</argument>
104-
<argument>--proto_path=${project.basedir}/src/main/proto</argument>
105-
<argument>${project.basedir}/src/main/proto/dtype.proto</argument>
106-
<argument>${project.basedir}/src/main/proto/scalar.proto</argument>
107-
<argument>${project.basedir}/src/main/proto/encodings.proto</argument>
108-
</arguments>
109-
</configuration>
110-
</execution>
111-
</executions>
112-
</plugin>
53+
To regenerate after editing .fbs or .proto schemas:
54+
brew install flatbuffers protobuf
55+
./mvnw generate-sources -pl core -P regenerate-sources
56+
Then commit the updated files.
57+
Any flatc version works — the profile strips the version guard automatically.
58+
-->
59+
<profiles>
60+
<profile>
61+
<id>regenerate-sources</id>
62+
<build>
63+
<plugins>
64+
<!-- 1. Generate Java from FlatBuffer schemas -->
65+
<plugin>
66+
<groupId>org.codehaus.mojo</groupId>
67+
<artifactId>exec-maven-plugin</artifactId>
68+
<executions>
69+
<execution>
70+
<id>flatc-generate</id>
71+
<phase>generate-sources</phase>
72+
<goals>
73+
<goal>exec</goal>
74+
</goals>
75+
<configuration>
76+
<executable>flatc</executable>
77+
<workingDirectory>${project.basedir}/src/main/flatbuffers</workingDirectory>
78+
<arguments>
79+
<argument>--java</argument>
80+
<argument>-I</argument>
81+
<argument>.</argument>
82+
<argument>-o</argument>
83+
<argument>${project.basedir}/src/main/java</argument>
84+
<argument>vortex-file/footer.fbs</argument>
85+
<argument>vortex-layout/layout.fbs</argument>
86+
<argument>vortex-array/array.fbs</argument>
87+
<argument>vortex-dtype/dtype.fbs</argument>
88+
</arguments>
89+
</configuration>
90+
</execution>
91+
<!-- 2. Generate Java from Protobuf schemas -->
92+
<execution>
93+
<id>protoc-generate</id>
94+
<phase>generate-sources</phase>
95+
<goals>
96+
<goal>exec</goal>
97+
</goals>
98+
<configuration>
99+
<executable>protoc</executable>
100+
<arguments>
101+
<argument>--java_out=${project.basedir}/src/main/java</argument>
102+
<argument>--proto_path=${project.basedir}/src/main/proto</argument>
103+
<argument>${project.basedir}/src/main/proto/dtype.proto</argument>
104+
<argument>${project.basedir}/src/main/proto/scalar.proto</argument>
105+
<argument>${project.basedir}/src/main/proto/encodings.proto</argument>
106+
</arguments>
107+
</configuration>
108+
</execution>
109+
</executions>
110+
</plugin>
113111

114-
<!-- 2b. Strip ValidateVersion() — flatc ties it to the exact release,
115-
but Maven Central lags so the constant won't exist in flatbuffers-java. -->
116-
<plugin>
117-
<groupId>org.apache.maven.plugins</groupId>
118-
<artifactId>maven-antrun-plugin</artifactId>
119-
<executions>
120-
<execution>
121-
<id>flatc-strip-version-guard</id>
122-
<phase>generate-sources</phase>
123-
<goals>
124-
<goal>run</goal>
125-
</goals>
126-
<configuration>
127-
<target>
128-
<replaceregexp
129-
match=" public static void ValidateVersion\(\) \{ Constants\.[^}]+\}&#10;"
130-
replace=""
131-
flags="g">
132-
<fileset
133-
dir="${project.build.directory}/generated-sources/flatbuffers/io/github/dfa1/vortex/fbs"
134-
includes="*.java"/>
135-
</replaceregexp>
136-
</target>
137-
</configuration>
138-
</execution>
139-
</executions>
140-
</plugin>
112+
<!--
113+
Why we strip ValidateVersion():
114+
flatc injects a `public static void ValidateVersion()` into every generated
115+
class that calls `Constants.FLATBUFFERS_25_X_Y()` — a method that only exists
116+
in the flatbuffers-java jar built from the *same* flatc release. This is a
117+
developer-experience guard: it prevents accidentally pairing a stale generated
118+
file with a newer runtime jar.
141119
142-
<!-- 3. Register generated-sources dirs for compilation -->
143-
<plugin>
144-
<groupId>org.codehaus.mojo</groupId>
145-
<artifactId>build-helper-maven-plugin</artifactId>
146-
<executions>
147-
<execution>
148-
<id>add-generated-sources</id>
149-
<phase>generate-sources</phase>
150-
<goals>
151-
<goal>add-source</goal>
152-
</goals>
153-
<configuration>
154-
<sources>
155-
<source>${project.build.directory}/generated-sources/flatbuffers</source>
156-
<source>${project.build.directory}/generated-sources/protobuf</source>
157-
</sources>
158-
</configuration>
159-
</execution>
160-
</executions>
161-
</plugin>
120+
The problem: flatbuffers-java on Maven Central lags the flatc CLI by months.
121+
As of 2026-06, Maven Central tops out at 25.2.10 while `brew install flatbuffers`
122+
installs 25.12.19. Any attempt to compile freshly generated code against the
123+
Maven Central jar fails with "cannot find symbol: Constants.FLATBUFFERS_25_12_19".
124+
125+
The strip is safe because:
126+
- The FlatBuffers binary wire format is stable across releases; 25.2.10 can
127+
read bytes written by code generated with any 25.x flatc.
128+
- We commit the stripped files, so the check is irrelevant at compile time for
129+
normal builds (no flatc involved).
130+
- The strip only runs inside the `regenerate-sources` profile, not during
131+
normal `./mvnw verify` builds.
132+
133+
Re-evaluate if flatbuffers-java is ever published to Maven Central in lock-step
134+
with flatc releases — at that point the antrun step can be removed.
135+
-->
136+
<plugin>
137+
<groupId>org.apache.maven.plugins</groupId>
138+
<artifactId>maven-antrun-plugin</artifactId>
139+
<executions>
140+
<execution>
141+
<id>flatc-strip-version-guard</id>
142+
<phase>generate-sources</phase>
143+
<goals>
144+
<goal>run</goal>
145+
</goals>
146+
<configuration>
147+
<target>
148+
<replaceregexp
149+
match=" public static void ValidateVersion\(\) \{ Constants\.[^}]+\}&#10;"
150+
replace=""
151+
flags="g">
152+
<fileset
153+
dir="${project.basedir}/src/main/java/io/github/dfa1/vortex/fbs"
154+
includes="*.java"/>
155+
</replaceregexp>
156+
</target>
157+
</configuration>
158+
</execution>
159+
</executions>
160+
</plugin>
161+
</plugins>
162+
</build>
163+
</profile>
164+
</profiles>
162165

163-
</plugins>
164-
</build>
165166
</project>

‎core/src/main/java/io/github/dfa1/vortex/encoding/PcoEncoding.java‎

Lines changed: 5 additions & 74 deletions
Original file line numberDiff line numberDiff line change
@@ -85,10 +85,6 @@ static final class Decoder {
8585

8686
private static final ValueLayout.OfLong LE_LONG =
8787
ValueLayout.JAVA_LONG_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
88-
private static final ValueLayout.OfInt LE_INT = PTypeIO.LE_INT;
89-
private static final ValueLayout.OfShort LE_SHORT = PTypeIO.LE_SHORT;
90-
private static final ValueLayout.OfFloat LE_FLOAT = PTypeIO.LE_FLOAT;
91-
private static final ValueLayout.OfDouble LE_DOUBLE = PTypeIO.LE_DOUBLE;
9288

9389
static Array decode(DecodeContext ctx) {
9490
EncodingProtos.PcoMetadata meta = parseMeta(ctx);
@@ -248,16 +244,12 @@ static Array decode(DecodeContext ctx) {
248244
}
249245
}
250246

251-
// Convert raw U-latents → typed values.
252-
// I64/U64: element size == 8 bytes == rawLatents slot size; convert in-place.
247+
// Convert raw U-latents → typed values; resize from 8-byte slots to elemBytes.
253248
int elemBytes = ptype.byteSize();
254-
MemorySegment compactOut;
255-
if (ptype == PType.I64 || ptype == PType.U64) {
256-
convertLatentsToCompact(rawLatents, validCount, ptype, rawLatents);
257-
compactOut = rawLatents;
258-
} else {
259-
compactOut = ctx.arena().allocate(validCount * elemBytes);
260-
convertLatentsToCompact(rawLatents, validCount, ptype, compactOut);
249+
MemorySegment compactOut = ctx.arena().allocate(validCount * elemBytes);
250+
for (long i = 0; i < validCount; i++) {
251+
long latent = rawLatents.get(LE_LONG, i * Long.BYTES);
252+
PTypeIO.set(compactOut, i * elemBytes, ptype, fromLatentOrdered(latent, ptype));
261253
}
262254

263255
if (validity == null) {
@@ -538,67 +530,6 @@ private static long predictConv1(MemorySegment seg, long baseOff, int pos, int o
538530
return (s >> quantization) & mask;
539531
}
540532

541-
/// Convert rawLatents (U64 per slot, 8 bytes each) → typed compactOut (elemBytes per slot).
542-
///
543-
/// Ptype switch is hoisted outside the loop so each case is a tight, JIT-friendly loop
544-
/// with direct FFM reads/writes and no MethodHandle dispatch per element.
545-
private static void convertLatentsToCompact(
546-
MemorySegment rawLatents, long n, PType ptype, MemorySegment compactOut) {
547-
switch (ptype) {
548-
case I64 -> {
549-
for (long i = 0; i < n; i++) {
550-
compactOut.setAtIndex(LE_LONG, i,
551-
rawLatents.getAtIndex(LE_LONG, i) ^ Long.MIN_VALUE);
552-
}
553-
}
554-
case U64 -> {
555-
for (long i = 0; i < n; i++) {
556-
compactOut.setAtIndex(LE_LONG, i, rawLatents.getAtIndex(LE_LONG, i));
557-
}
558-
}
559-
case I32 -> {
560-
for (long i = 0; i < n; i++) {
561-
compactOut.setAtIndex(LE_INT, i,
562-
(int) (rawLatents.getAtIndex(LE_LONG, i) ^ 0x80000000L));
563-
}
564-
}
565-
case U32 -> {
566-
for (long i = 0; i < n; i++) {
567-
compactOut.setAtIndex(LE_INT, i,
568-
(int) rawLatents.getAtIndex(LE_LONG, i));
569-
}
570-
}
571-
case I16 -> {
572-
for (long i = 0; i < n; i++) {
573-
compactOut.setAtIndex(LE_SHORT, i,
574-
(short) (rawLatents.getAtIndex(LE_LONG, i) ^ 0x8000L));
575-
}
576-
}
577-
case U16 -> {
578-
for (long i = 0; i < n; i++) {
579-
compactOut.setAtIndex(LE_SHORT, i,
580-
(short) rawLatents.getAtIndex(LE_LONG, i));
581-
}
582-
}
583-
case F32 -> {
584-
for (long i = 0; i < n; i++) {
585-
long l32 = rawLatents.getAtIndex(LE_LONG, i) & 0xFFFFFFFFL;
586-
int bits = (int) ((l32 & 0x80000000L) != 0 ? l32 ^ 0x80000000L : l32 ^ 0xFFFFFFFFL);
587-
compactOut.setAtIndex(LE_FLOAT, i, Float.intBitsToFloat(bits));
588-
}
589-
}
590-
case F64 -> {
591-
for (long i = 0; i < n; i++) {
592-
long latent = rawLatents.getAtIndex(LE_LONG, i);
593-
long bits = (latent & Long.MIN_VALUE) != 0 ? latent ^ Long.MIN_VALUE : ~latent;
594-
compactOut.setAtIndex(LE_DOUBLE, i, Double.longBitsToDouble(bits));
595-
}
596-
}
597-
default -> throw new VortexException(EncodingId.VORTEX_PCO,
598-
"pco: unsupported ptype in convertLatentsToCompact: " + ptype);
599-
}
600-
}
601-
602533
/// Inverse of pcodec {@code to_latent_ordered}: maps raw U-latent back to typed bits.
603534
private static long fromLatentOrdered(long latent, PType ptype) {
604535
return switch (ptype) {

0 commit comments

Comments
 (0)