Skip to content

Commit cc5f519

Browse files
committed
Merge remote-tracking branch 'origin/main' into mt_qwp-cleanup-params
# Conflicts: # core/src/main/java/io/questdb/client/QuestDBBuilder.java
2 parents 99e2ecd + 236ea59 commit cc5f519

54 files changed

Lines changed: 5271 additions & 195 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/ci.yml

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
name: CI
2+
3+
on:
4+
pull_request:
5+
push:
6+
branches:
7+
- main
8+
9+
concurrency:
10+
group: ci-${{ github.workflow }}-${{ github.ref }}
11+
cancel-in-progress: true
12+
13+
defaults:
14+
run:
15+
shell: bash
16+
17+
jobs:
18+
# JDK 8 is the source of truth: the client ships as a Java 8 artifact
19+
# (io.questdb:questdb-client) and is released from JDK 8, so on JDK 8 it must
20+
# compile, the full test suite must pass against the committed native
21+
# libraries, and the javadoc jar must build (-P javadoc attaches it at the
22+
# package phase). The committed native .so/.dylib/.dll are enough -- the only
23+
# git submodule (zstd) is needed solely for C++ native rebuilds, not here.
24+
build-jdk8:
25+
name: Build, test & javadoc (JDK 8)
26+
runs-on: ubuntu-latest
27+
timeout-minutes: 45
28+
steps:
29+
- name: Check out
30+
uses: actions/checkout@v4
31+
32+
- name: Set up JDK 8
33+
uses: actions/setup-java@v4
34+
with:
35+
distribution: temurin
36+
java-version: "8"
37+
cache: maven
38+
39+
- name: Compile, test, and build javadoc
40+
run: mvn -B -ntp -P javadoc clean install
41+
42+
# The client is also consumed as a submodule of the main questdb repo, which
43+
# builds on JDK 25. Guard against JDK 25 compile breakage (main + test
44+
# sources, both modules) and confirm the javadoc jar builds on JDK 25 too
45+
# (-P javadoc attaches it at the package phase). Do NOT run the tests -- the
46+
# parent repo runs them against a real server.
47+
compile-jdk25:
48+
name: Compile & javadoc smoke (JDK 25)
49+
runs-on: ubuntu-latest
50+
timeout-minutes: 20
51+
steps:
52+
- name: Check out
53+
uses: actions/checkout@v4
54+
55+
- name: Set up JDK 25
56+
uses: actions/setup-java@v4
57+
with:
58+
distribution: temurin
59+
java-version: "25"
60+
cache: maven
61+
62+
- name: Compile (main + test) and build javadoc (no tests run)
63+
run: mvn -B -ntp -P javadoc -DskipTests clean package

CLAUDE.md

Lines changed: 30 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,30 @@ Guidance for Claude Code when working with the QuestDB Java client
77

88
The standalone client repo for QuestDB ingestion (legacy ILP and QWP). It is
99
a Git submodule of the `questdb` repository and ships as the
10-
`io.questdb:questdb-client` Maven artifact. The module targets Java 11, so
11-
only legacy Java features are available here — no enhanced switch, no
12-
multiline strings, no pattern variables in `instanceof`.
10+
`io.questdb:questdb-client` Maven artifact.
11+
12+
**Dual JDK target — always aim at both.** The client ships as a Java 8
13+
artifact (JDK 8 is the source of truth: CI builds, tests, and releases on
14+
JDK 8) but must also compile cleanly on JDK 11+ (CI runs a JDK 25
15+
compile/javadoc smoke job). The right Maven profile (`java8` /
16+
`java11+`) auto-activates from the running JDK; a `src/main/java8` vs
17+
`src/main/java11` source root supplies the per-JDK shims. Practically this
18+
means **Java 8 is the language/API floor** — every change must build on
19+
both, so write to Java 8 and never introduce an API or syntax that only a
20+
newer JDK has.
21+
22+
Disallowed here (newer than Java 8), among others:
23+
- `var` local-variable type inference (10), enhanced `switch` (14),
24+
multiline text blocks (15), pattern variables in `instanceof` (16)
25+
- `List.of` / `List.copyOf` / `Map.of` and friends (9/10),
26+
`String.repeat` / `String.strip` / `String.isBlank` (11)
27+
- `ProcessHandle` (9), `Thread.onSpinWait` (9),
28+
the `Provider(String, String, String)` ctor (9 — use the Java 8
29+
`Provider(String, double, String)` overload)
30+
31+
When in doubt, check the API's `@since` and keep it `<= 1.8`. JDK
32+
9+-only behaviour belongs behind the `src/main/java11` shim with a Java 8
33+
counterpart in `src/main/java8`.
1334

1435
## Git & PR Conventions
1536

@@ -35,13 +56,18 @@ multiline strings, no pattern variables in `instanceof`.
3556

3657
## Build
3758

38-
Standard Maven build. Targets Java 11:
59+
Standard Maven build. Dual-target (Java 8 floor, Java 11+ also supported);
60+
the `java8` / `java11+` profile auto-activates from the JDK running Maven:
3961

4062
```bash
4163
mvn clean install -DskipTests # build + install to local Maven cache
4264
mvn -pl core test # run client unit tests
4365
```
4466

67+
Build on a JDK 8 to validate the shipping artifact (the source-of-truth
68+
target); also confirm it compiles on a modern JDK (11+) before merging, the
69+
same two fronts CI guards.
70+
4571
The parent `questdb` repo's `local-client` profile pulls this module as a
4672
sub-module so server changes can build against unpublished client code; if
4773
you change client code, install it (or pass `-P local-client` in the parent)

core/pom.xml

Lines changed: 78 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,10 @@
3737
<test.exclude>None</test.exclude>
3838
<test.include>%regex[.*[^o].class]</test.include><!-- exclude module-info.class-->
3939
<jmh.version>1.37</jmh.version>
40+
<!-- JDK-version-specific source root; overridden in the java8 profile -->
41+
<compat.src.dir>src/main/java11</compat.src.dir>
42+
<!-- logback 1.5.x needs Java 11; the java8 profile drops to the 1.3.x line -->
43+
<logback.version>1.5.25</logback.version>
4044
</properties>
4145

4246
<version>1.3.5-SNAPSHOT</version>
@@ -184,6 +188,27 @@
184188
</excludes>
185189
</configuration>
186190
</plugin>
191+
<!-- Adds the JDK-version-specific source root (src/main/java11 or src/main/java8)
192+
so the FDBigInteger bridge / module-access shim compiles on each JDK. -->
193+
<plugin>
194+
<groupId>org.codehaus.mojo</groupId>
195+
<artifactId>build-helper-maven-plugin</artifactId>
196+
<version>3.6.0</version>
197+
<executions>
198+
<execution>
199+
<id>add-compat-source</id>
200+
<phase>generate-sources</phase>
201+
<goals>
202+
<goal>add-source</goal>
203+
</goals>
204+
<configuration>
205+
<sources>
206+
<source>${compat.src.dir}</source>
207+
</sources>
208+
</configuration>
209+
</execution>
210+
</executions>
211+
</plugin>
187212
</plugins>
188213
</build>
189214

@@ -206,11 +231,11 @@
206231
</executions>
207232
<configuration>
208233
<doclint>none</doclint>
209-
<source>11</source>
234+
<source>${javac.compile.source}</source>
210235
<detectJavaApiLink>false</detectJavaApiLink>
211236
<additionalJOptions>
212-
<additionalJOption>${compilerArg1}</additionalJOption>
213-
<additionalJOption>${compilerArg2}</additionalJOption>
237+
<additionalJOption>${javadocJOption1}</additionalJOption>
238+
<additionalJOption>${javadocJOption2}</additionalJOption>
214239
</additionalJOptions>
215240
<sourceFileExcludes>
216241
<sourceFileExclude>${excludePattern1}</sourceFileExclude>
@@ -338,11 +363,11 @@
338363
</executions>
339364
<configuration>
340365
<doclint>none</doclint>
341-
<source>11</source>
366+
<source>${javac.compile.source}</source>
342367
<detectJavaApiLink>false</detectJavaApiLink>
343368
<additionalJOptions>
344-
<additionalJOption>${compilerArg1}</additionalJOption>
345-
<additionalJOption>${compilerArg2}</additionalJOption>
369+
<additionalJOption>${javadocJOption1}</additionalJOption>
370+
<additionalJOption>${javadocJOption2}</additionalJOption>
346371
</additionalJOptions>
347372
<sourceFileExcludes>
348373
<sourceFileExclude>${excludePattern1}</sourceFileExclude>
@@ -444,6 +469,11 @@
444469
<questdb.artifactid>questdb</questdb.artifactid>
445470
<compilerArg1>--add-exports</compilerArg1>
446471
<compilerArg2>java.base/jdk.internal.math=io.questdb.client</compilerArg2>
472+
<!-- javadoc needs the same module export to resolve jdk.internal.math.FDBigInteger
473+
referenced by the src/main/java11 FdBig bridge. Kept separate from the javac
474+
compilerArgs because javadoc and javac accept different option sets. -->
475+
<javadocJOption1>--add-exports</javadocJOption1>
476+
<javadocJOption2>java.base/jdk.internal.math=io.questdb.client</javadocJOption2>
447477
<excludePattern1>nothing-to-exclude-dummy-value-include-all-java11plus</excludePattern1>
448478
<excludeTestPattern1>nothing-to-exclude-dummy-value-include-all-java11plus</excludeTestPattern1>
449479
<javac.compile.source>${javac.target}</javac.compile.source>
@@ -466,6 +496,47 @@
466496
</dependency>
467497
</dependencies>
468498
</profile>
499+
<profile>
500+
<id>java8</id>
501+
<properties>
502+
<jdk.version>8</jdk.version>
503+
<java.enforce.version>[1.8,11)</java.enforce.version>
504+
<questdb.artifactid>questdb</questdb.artifactid>
505+
<!-- JDK8 javac rejects the add-exports flag; use a harmless no-op flag instead -->
506+
<compilerArg1>-Xlint:none</compilerArg1>
507+
<compilerArg2>-Xlint:none</compilerArg2>
508+
<!-- JDK8 javadoc rejects javac's -Xlint and needs no module export (the java8 FdBig
509+
bridge uses the open sun.misc.FDBigInteger). Pass a harmless no-op instead. -->
510+
<javadocJOption1>-quiet</javadocJOption1>
511+
<javadocJOption2>-quiet</javadocJOption2>
512+
<!-- module-info.java cannot be compiled on JDK8; exclude it from both the main
513+
(<excludes> = excludeTestPattern1) and test (<testExcludes> = excludePattern1)
514+
source sets. -->
515+
<excludePattern1>**/module-info.java</excludePattern1>
516+
<excludeTestPattern1>**/module-info.java</excludeTestPattern1>
517+
<compat.src.dir>src/main/java8</compat.src.dir>
518+
<javac.compile.source>1.8</javac.compile.source>
519+
<javac.compile.target>1.8</javac.compile.target>
520+
<!-- logback 1.5.x requires Java 11; the 1.3.x line is the Java 8 compatible branch -->
521+
<logback.version>1.3.15</logback.version>
522+
</properties>
523+
<activation>
524+
<jdk>1.8</jdk>
525+
</activation>
526+
<dependencies>
527+
<dependency>
528+
<groupId>org.jetbrains</groupId>
529+
<artifactId>annotations</artifactId>
530+
<version>17.0.0</version>
531+
<scope>provided</scope>
532+
</dependency>
533+
<dependency>
534+
<groupId>org.slf4j</groupId>
535+
<artifactId>slf4j-api</artifactId>
536+
<version>2.0.17</version>
537+
</dependency>
538+
</dependencies>
539+
</profile>
469540
</profiles>
470541

471542
<dependencies>
@@ -485,7 +556,7 @@
485556
<dependency>
486557
<groupId>ch.qos.logback</groupId>
487558
<artifactId>logback-classic</artifactId>
488-
<version>1.5.25</version>
559+
<version>${logback.version}</version>
489560
<scope>test</scope>
490561
</dependency>
491562
<dependency>

core/src/main/java/io/questdb/client/QuestDBBuilder.java

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -90,7 +90,15 @@ public QuestDBBuilder acquireTimeoutMillis(long millis) {
9090
* front -- so a malformed config fails here even when both pools have
9191
* {@code min == 0} and nothing connects -- then eagerly creates {@code min}
9292
* connections in each pool; further slots are allocated lazily up to
93-
* {@code max} when load demands and reaped back to {@code min} when idle.
93+
* {@code max} when load demands and reaped back to {@code min} when
94+
* idle.
95+
* <p>
96+
* Non-blocking on startup recovery: when store-and-forward is enabled,
97+
* unacked data a previous run left in this pool's managed slots is
98+
* recovered on a background housekeeper thread shortly after this method
99+
* returns -- so {@code build()} does not block on a slow or
100+
* reachable-but-not-acking server. The recovered data is durable on disk
101+
* and is delivered once the server acks; until then it stays preserved.
94102
*/
95103
public QuestDB build() {
96104
if (ingestConfig == null) {

core/src/main/java/io/questdb/client/Sender.java

Lines changed: 84 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1025,6 +1025,15 @@ final class LineSenderBuilder {
10251025
// runtime lands in a follow-up commit. For now we surface the
10261026
// count via logging so users can confirm orphans are being seen.
10271027
private boolean drainOrphans = false;
1028+
// Orphan-scan exclusion for the connection pool. The pool co-manages
1029+
// exactly <orphanDrainBase>-<i> for i in [0, orphanDrainSlotCount) and
1030+
// recovers each of those on (re)creation, so pooled senders must never
1031+
// treat one another's live slots as drainable orphans. Anything else --
1032+
// a different base, a bare un-suffixed id, OR a same-base index at or
1033+
// above the count (a slot left behind by a larger pool before maxSize
1034+
// shrank) -- is still drained, so unacked data is never stranded.
1035+
private String orphanDrainBase;
1036+
private int orphanDrainSlotCount;
10281037
private long durableAckKeepaliveIntervalMillis = DURABLE_ACK_KEEPALIVE_NOT_SET;
10291038
// Optional user-supplied async error handler. When null, the sender
10301039
// uses DefaultSenderErrorHandler.INSTANCE (loud-not-silent log).
@@ -1469,7 +1478,15 @@ public Sender build() {
14691478
} else {
14701479
if (!Files.exists(sfDir)) {
14711480
int rc = Files.mkdir(sfDir, Files.DIR_MODE_DEFAULT);
1472-
if (rc != 0) {
1481+
// mkdir is non-zero on failure, but "already exists"
1482+
// is one such failure. Multiple SF senders sharing one
1483+
// sf_dir can be built concurrently (the pool calls
1484+
// build() outside its lock), so two threads can both
1485+
// pass the exists() check and race into mkdir; the
1486+
// loser gets EEXIST. Treat a benign creation race --
1487+
// the dir now exists -- as success and only fail when
1488+
// the directory is genuinely absent afterwards.
1489+
if (rc != 0 && !Files.exists(sfDir)) {
14731490
throw new LineSenderException(
14741491
"could not create sf_dir: " + sfDir + " rc=" + rc);
14751492
}
@@ -1544,7 +1561,7 @@ public Sender build() {
15441561
if (drainOrphans && sfDir != null) {
15451562
io.questdb.client.std.ObjList<String> orphans =
15461563
io.questdb.client.cutlass.qwp.client.sf.cursor.OrphanScanner
1547-
.scan(sfDir, senderId);
1564+
.scan(sfDir, senderId, orphanDrainBase, orphanDrainSlotCount);
15481565
if (orphans.size() > 0) {
15491566
org.slf4j.LoggerFactory.getLogger(LineSenderBuilder.class)
15501567
.info("dispatching drainers for {} orphan slot(s) under {} "
@@ -2459,6 +2476,71 @@ public LineSenderBuilder senderId(String id) {
24592476
return this;
24602477
}
24612478

2479+
/**
2480+
* The slot id ({@code sender_id}) currently configured on this
2481+
* builder, either parsed from the config string or left at its
2482+
* {@code "default"} default. Introspection hook for the connection
2483+
* pool, which derives a distinct per-slot id from this base so that
2484+
* multiple pooled senders sharing one {@code sf_dir} don't collide
2485+
* on the slot {@code flock}.
2486+
*/
2487+
public String getConfiguredSenderId() {
2488+
return senderId;
2489+
}
2490+
2491+
/**
2492+
* The store-and-forward group root ({@code sf_dir}) currently
2493+
* configured on this builder, or {@code null} when SF is disabled.
2494+
* Introspection hook for the connection pool, which needs the group
2495+
* root to locate its own managed slot dirs {@code <sf_dir>/<base>-<i>}
2496+
* when recovering unacked data a previous run left behind.
2497+
*/
2498+
public String getConfiguredSfDir() {
2499+
return sfDir;
2500+
}
2501+
2502+
/**
2503+
* Excludes the connection pool's <em>live</em> slot set from
2504+
* {@link #drainOrphans(boolean)} scanning: a sibling slot under
2505+
* {@code sf_dir} named {@code <base>-<index>} with
2506+
* {@code 0 <= index < slotCount} is never treated as a drainable orphan.
2507+
* <p>
2508+
* Internal introspection hook for the connection pool. The pool gives
2509+
* each pooled SF sender a distinct slot id {@code <base>-<index>} and
2510+
* recovers each slot's unacked data itself when it (re)creates that
2511+
* slot. Without this exclusion, one pooled sender's startup drainer
2512+
* could adopt a sibling pool slot's lock and dir, reintroducing the
2513+
* very "sf slot already in use" collision the per-slot ids were added
2514+
* to prevent.
2515+
* <p>
2516+
* Unlike a blanket {@code <base>-} prefix exclusion, the bound is the
2517+
* pool's {@code maxSize}: a same-base slot whose index is at or above
2518+
* {@code slotCount} (e.g. {@code <base>-3} left behind by a larger pool
2519+
* before {@code maxSize} shrank from 4 to 2) is NOT excluded and is
2520+
* drained like any foreign leftover, so its unacked data is recovered
2521+
* instead of being silently stranded. Foreign leftovers (a different
2522+
* base, or a bare un-suffixed id) are also still drained.
2523+
* <p>
2524+
* Pass a {@code null}/empty base or {@code slotCount <= 0} to disable
2525+
* the exclusion (the default).
2526+
*/
2527+
public LineSenderBuilder orphanDrainExcludeManagedSlots(String base, int slotCount) {
2528+
this.orphanDrainBase = base;
2529+
this.orphanDrainSlotCount = slotCount;
2530+
return this;
2531+
}
2532+
2533+
/**
2534+
* True iff store-and-forward is enabled (an {@code sf_dir} was set).
2535+
* Introspection hook for the connection pool: SF senders own an
2536+
* exclusive on-disk slot, so each pooled sender needs its own slot
2537+
* id, whereas non-SF (memory-mode / HTTP / TCP) senders share no
2538+
* such resource and need no per-slot identity.
2539+
*/
2540+
public boolean isStoreAndForwardEnabled() {
2541+
return sfDir != null;
2542+
}
2543+
24622544
/**
24632545
* Per-call deadline for {@code Sender.flush()} spinning on a full
24642546
* cursor segment ring waiting for ACKs to drain space. Default

0 commit comments

Comments
 (0)