Skip to content

Commit e0ebdf0

Browse files
bluestreak01bluestreak01
committed
test(qwp): close the C-2 durability test gaps with mutant-killer coverage
A live mutation accident proved the suite blind to durability-critical code: 88d6b79 shipped requestSyncBeforeRotation neutralized ('return false; // MUTANT: gate neutralized') and 369 tests stayed green; only human re-reading caught it (71cfbe2). Re-running that mutant against today's suite still passed 385/385, as did deleting SfManifest.update's monotonic clamp. Every gap below is now covered, and both mutants were re-applied after writing the tests to verify they die. - SegmentManagerPeriodicSyncTest.testRotationGateDefersRotationUntil- PredecessorDurable: kills the gate mutant at three points (rotation must backpressure while the predecessor is non-durable, the gate's sync request must run the barrier before the interval deadline, the retried append must rotate). Verified: mutant now fails 'expected -1 but was 2'. - SegmentManagerPeriodicSyncTest.testSyncPassStopsAtFirstFailureThen- RetryCoversAllSegments: >= 2 non-durable live segments (recovered segments start non-durable) prove the pass aborts at the first barrier failure and the healed retry covers every segment before the producer unlatches. - SfManifestClampTest: direct pin on the update() monotonic clamp, including independent per-field clamping and durable persistence across reopen. Verified: clamp-deletion mutant now fails 'expected 10 but was 5'. SfManifest and the five members under test are widened to public within the already-exported internal package (JPMS forbids same-package test classes; matches MmapSegment/SegmentRing). - CursorSendEngineTest.testCheckDurabilitySurfacesLatchedFailureTo- SenderEntryPoints: pins the engine delegation seam behind flush()/awaitAckedFsn() (zero references before): quiet when clean, throws the latched instance repeatedly until cleared. - SegmentManagerCloseRaceTest.testSecondBoundedJoinReapsWorker- FinishingExitCleanups: first join times out against a worker parked in its exit cleanups (workerLoopExited set before cleanups run), the second fixed-budget join must reap and close() must free the scratch itself. Every existing close-race test only reached the first join. - CursorSendEngineClosePartialEnumerationTest: torn close-time directory enumeration (findNext < 0 mid-walk, injected via a FilesFacade proxy) must drive ZERO unlinks, keep the watermark, and a successor's fully-drained close retries the cleanup. The sibling unlink-failure test only covered failing unlinks via the root-skipped permission trick. - SourceHygieneTest: tripwire that fails the build on 'MUTANT' markers in main sources -- the exact artifact 88d6b79 shipped. sf package: 392 tests green (385 + 7 new).
1 parent db8938e commit e0ebdf0

7 files changed

Lines changed: 691 additions & 6 deletions

File tree

‎core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SfManifest.java‎

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -38,8 +38,13 @@
3838
* update. Recovery selects the valid record with the greatest generation.
3939
* The separate 4 KiB slots prevent one aligned 512-byte or 4 KiB sector tear
4040
* from erasing both the update and the previous committed boundary.
41+
* <p>
42+
* Public within this exported-internal package (like {@link MmapSegment} and
43+
* {@link SegmentRing}) so the boundary contract -- notably the monotonic
44+
* clamp in {@link #update} -- can be pinned by direct unit tests; it is not
45+
* part of the supported client API.
4146
*/
42-
final class SfManifest implements QuietCloseable {
47+
public final class SfManifest implements QuietCloseable {
4348
static final String FILE_NAME = "sf-manifest.bin";
4449
private static final Logger LOG = LoggerFactory.getLogger(SfManifest.class);
4550
private static final int CRC_OFFSET = 60;
@@ -71,7 +76,7 @@ private SfManifest(FilesFacade filesFacade, String path, int fd,
7176
this.writeScratch = Unsafe.malloc(RECORD_SIZE, MemoryTag.NATIVE_DEFAULT);
7277
}
7378

74-
static SfManifest create(FilesFacade filesFacade, String dir, long headBase, long activeBase) {
79+
public static SfManifest create(FilesFacade filesFacade, String dir, long headBase, long activeBase) {
7580
String path = dir + "/" + FILE_NAME;
7681
int fd = filesFacade.openRWExclusive(path);
7782
if (fd < 0) {
@@ -104,7 +109,7 @@ static SfManifest create(FilesFacade filesFacade, String dir, long headBase, lon
104109
}
105110
}
106111

107-
static SfManifest open(FilesFacade filesFacade, String dir) {
112+
public static SfManifest open(FilesFacade filesFacade, String dir) {
108113
String path = dir + "/" + FILE_NAME;
109114
if (!filesFacade.exists(path)) {
110115
return null;
@@ -168,7 +173,7 @@ static SfManifest open(FilesFacade filesFacade, String dir) {
168173
}
169174
}
170175

171-
long activeBase() {
176+
public long activeBase() {
172177
return activeBase;
173178
}
174179

@@ -184,7 +189,7 @@ public synchronized void close() {
184189
}
185190
}
186191

187-
long headBase() {
192+
public long headBase() {
188193
return headBase;
189194
}
190195

@@ -199,7 +204,7 @@ static boolean removeFile(FilesFacade filesFacade, String dir) {
199204
return filesFacade.remove(path) || !filesFacade.exists(path);
200205
}
201206

202-
synchronized void update(long newHeadBase, long newActiveBase) {
207+
public synchronized void update(long newHeadBase, long newActiveBase) {
203208
if (closed) {
204209
throw new IllegalStateException("SF manifest is closed");
205210
}

‎core/src/test/java/io/questdb/client/test/SourceHygieneTest.java‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
/*******************************************************************************
2+
* ___ _ ____ ____
3+
* / _ \ _ _ ___ ___| |_| _ \| __ )
4+
* | | | | | | |/ _ \/ __| __| | | | _ \
5+
* | |_| | |_| | __/\__ \ |_| |_| | |_) |
6+
* \__\_\\__,_|\___||___/\__|____/|____/
7+
*
8+
* Copyright (c) 2014-2019 Appsicle
9+
* Copyright (c) 2019-2026 QuestDB
10+
*
11+
* Licensed under the Apache License, Version 2.0 (the "License");
12+
* you may not use this file except in compliance with the License.
13+
* You may obtain a copy of the License at
14+
*
15+
* http://www.apache.org/licenses/LICENSE-2.0
16+
*
17+
* Unless required by applicable law or agreed to in writing, software
18+
* distributed under the License is distributed on an "AS IS" BASIS,
19+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
20+
* See the License for the specific language governing permissions and
21+
* limitations under the License.
22+
*
23+
******************************************************************************/
24+
25+
package io.questdb.client.test;
26+
27+
import org.junit.Test;
28+
29+
import java.io.IOException;
30+
import java.nio.charset.StandardCharsets;
31+
import java.nio.file.FileVisitResult;
32+
import java.nio.file.Files;
33+
import java.nio.file.Path;
34+
import java.nio.file.Paths;
35+
import java.nio.file.SimpleFileVisitor;
36+
import java.nio.file.attribute.BasicFileAttributes;
37+
import java.util.ArrayList;
38+
import java.util.List;
39+
40+
import static org.junit.Assert.assertTrue;
41+
42+
/**
43+
* Tripwire against mutation-tooling edits leaking into shipped sources.
44+
* <p>
45+
* Commit {@code 88d6b792} accidentally captured a concurrent mutation-testing
46+
* edit that neutralized {@code SegmentRing.requestSyncBeforeRotation} to
47+
* {@code return false; // MUTANT: gate neutralized} -- injected into a shared
48+
* worktree between test validation and {@code git add}. The full suite stayed
49+
* green and only human re-reading caught it ({@code 71cfbe2e}). Mutation
50+
* tools mark their edits precisely so they can be found; this test makes that
51+
* marker a build failure instead of a code-review lottery ticket.
52+
*/
53+
public class SourceHygieneTest {
54+
55+
private static final String MARKER = "MUTANT";
56+
57+
@Test
58+
public void testNoMutationToolMarkersInMainSources() throws IOException {
59+
// Surefire runs with the module directory (core/) as cwd.
60+
Path root = Paths.get("src", "main", "java");
61+
if (!Files.isDirectory(root)) {
62+
root = Paths.get("core", "src", "main", "java");
63+
}
64+
assertTrue("main source root not found from " + Paths.get("").toAbsolutePath()
65+
+ " -- fix the path resolution rather than skipping the tripwire",
66+
Files.isDirectory(root));
67+
68+
final List<String> offenders = new ArrayList<>();
69+
Files.walkFileTree(root, new SimpleFileVisitor<Path>() {
70+
@Override
71+
public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {
72+
if (file.toString().endsWith(".java")) {
73+
List<String> lines = Files.readAllLines(file, StandardCharsets.UTF_8);
74+
for (int i = 0; i < lines.size(); i++) {
75+
if (lines.get(i).contains(MARKER)) {
76+
offenders.add(file + ":" + (i + 1) + " " + lines.get(i).trim());
77+
}
78+
}
79+
}
80+
return FileVisitResult.CONTINUE;
81+
}
82+
});
83+
assertTrue("mutation-tool markers must never reach main sources "
84+
+ "(a neutralized durability gate shipped exactly this way in 88d6b792); "
85+
+ "offending lines:\n" + String.join("\n", offenders),
86+
offenders.isEmpty());
87+
}
88+
}

‎core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/CursorSendEngineClosePartialEnumerationTest.java‎

Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
/*******************************************************************************
2+
* ___ _ ____ ____
3+
* / _ \ _ _ ___ ___| |_| _ \| __ )
4+
* | | | | | | |/ _ \/ __| __| | | | _ \
5+
* | |_| | |_| | __/\__ \ |_| |_| | |_) |
6+
* \__\_\\__,_|\___||___/\__|____/|____/
7+
*
8+
* Copyright (c) 2014-2019 Appsicle
9+
* Copyright (c) 2019-2026 QuestDB
10+
*
11+
* Licensed under the Apache License, Version 2.0 (the "License");
12+
* you may not use this file except in compliance with the License.
13+
* You may obtain a copy of the License at
14+
*
15+
* http://www.apache.org/licenses/LICENSE-2.0
16+
*
17+
* Unless required by applicable law or agreed to in writing, software
18+
* distributed under the License is distributed on an "AS IS" BASIS,
19+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
20+
* See the License for the specific language governing permissions and
21+
* limitations under the License.
22+
*
23+
******************************************************************************/
24+
25+
package io.questdb.client.test.cutlass.qwp.client.sf.cursor;
26+
27+
import io.questdb.client.cutlass.qwp.client.sf.cursor.CursorSendEngine;
28+
import io.questdb.client.cutlass.qwp.client.sf.cursor.SegmentManager;
29+
import io.questdb.client.std.Files;
30+
import io.questdb.client.std.FilesFacade;
31+
import io.questdb.client.std.MemoryTag;
32+
import io.questdb.client.std.Unsafe;
33+
import io.questdb.client.test.tools.TestUtils;
34+
import org.junit.After;
35+
import org.junit.Assert;
36+
import org.junit.Before;
37+
import org.junit.Test;
38+
39+
import java.lang.reflect.InvocationTargetException;
40+
import java.lang.reflect.Proxy;
41+
import java.util.concurrent.atomic.AtomicBoolean;
42+
import java.util.concurrent.atomic.AtomicInteger;
43+
44+
/**
45+
* Close-time unlink under a TORN directory enumeration ({@code findNext}
46+
* fails after the listing already produced entries). A partial listing must
47+
* not drive any unlink: removing only the files the walk happened to see
48+
* could delete the segment holding the highest frame while a lower one
49+
* survives, leaving residual state the retained ack watermark can no longer
50+
* vouch for. The contract is all-or-nothing: abort the cleanup, keep every
51+
* {@code .sfa} file and the watermark, and let the next recovery (or a
52+
* successor's fully-drained close) retry.
53+
* <p>
54+
* The sibling {@link CursorSendEngineCloseUnlinkFailureTest} injects a
55+
* failing UNLINK (permission trick, root-skipped); nothing exercised the
56+
* enumeration-abort branch itself. Same determinism trick as the sibling:
57+
* the shared manager is never started, so no worker touches the slot and no
58+
* concurrent enumeration can trip the armed fault.
59+
*/
60+
public class CursorSendEngineClosePartialEnumerationTest {
61+
62+
private String tmpDir;
63+
64+
@Before
65+
public void setUp() {
66+
tmpDir = TestUtils.createTmpDir("qdb-engine-close-enum-fault-");
67+
}
68+
69+
@After
70+
public void tearDown() {
71+
TestUtils.removeTmpDir(tmpDir);
72+
}
73+
74+
@Test(timeout = 20_000L)
75+
public void testTornCloseTimeEnumerationUnlinksNothingAndSuccessorRetries() throws Exception {
76+
TestUtils.assertMemoryLeak(() -> {
77+
final AtomicBoolean failFindNext = new AtomicBoolean();
78+
final AtomicInteger sfaRemoveAttempts = new AtomicInteger();
79+
FilesFacade faultFacade = (FilesFacade) Proxy.newProxyInstance(
80+
FilesFacade.class.getClassLoader(),
81+
new Class<?>[]{FilesFacade.class},
82+
(proxy, method, args) -> {
83+
if (failFindNext.get() && "findNext".equals(method.getName())) {
84+
return -1;
85+
}
86+
if ("remove".equals(method.getName())
87+
&& args != null && args.length == 1
88+
&& args[0] instanceof String
89+
&& ((String) args[0]).endsWith(".sfa")) {
90+
sfaRemoveAttempts.incrementAndGet();
91+
}
92+
try {
93+
return method.invoke(FilesFacade.INSTANCE, args);
94+
} catch (InvocationTargetException e) {
95+
throw e.getCause();
96+
}
97+
});
98+
99+
long segSize = 4096L;
100+
String slot = tmpDir + "/slot";
101+
Assert.assertEquals(0, Files.mkdir(slot, Files.DIR_MODE_DEFAULT));
102+
SegmentManager manager = new SegmentManager(
103+
segSize,
104+
SegmentManager.DEFAULT_POLL_NANOS,
105+
segSize * 4L,
106+
faultFacade,
107+
System::nanoTime);
108+
long buf = Unsafe.malloc(16, MemoryTag.NATIVE_DEFAULT);
109+
try {
110+
CursorSendEngine engine = new CursorSendEngine(slot, segSize, manager);
111+
boolean engineClosed = false;
112+
try {
113+
Assert.assertEquals(0L, engine.appendBlocking(buf, 16));
114+
Assert.assertTrue(engine.acknowledge(0L));
115+
116+
// Fully drained close, but the directory listing tears
117+
// mid-walk: the cleanup must abort BEFORE the first
118+
// unlink.
119+
failFindNext.set(true);
120+
engine.close();
121+
engineClosed = true;
122+
Assert.assertTrue("close must complete despite the aborted cleanup",
123+
engine.isCloseCompleted());
124+
Assert.assertEquals(
125+
"a torn enumeration must drive ZERO segment unlinks -- "
126+
+ "removing only the files the walk happened to see can "
127+
+ "strand residual state the watermark cannot vouch for",
128+
0, sfaRemoveAttempts.get());
129+
Assert.assertTrue("the segment file must survive the aborted cleanup",
130+
Files.exists(slot + "/sf-initial.sfa"));
131+
} finally {
132+
if (!engineClosed) {
133+
engine.close();
134+
}
135+
}
136+
137+
// Heal the directory walk; a successor adopts the slot,
138+
// recovers the residual (fully acknowledged) state, and its
139+
// own fully-drained close retries the cleanup successfully.
140+
failFindNext.set(false);
141+
CursorSendEngine successor = new CursorSendEngine(slot, segSize, manager);
142+
boolean successorClosed = false;
143+
try {
144+
Assert.assertTrue("successor must recover the residual slot state",
145+
successor.wasRecoveredFromDisk());
146+
successor.close();
147+
successorClosed = true;
148+
Assert.assertTrue(successor.isCloseCompleted());
149+
Assert.assertFalse(
150+
"successor's fully-drained close must retry and complete the unlink",
151+
Files.exists(slot + "/sf-initial.sfa"));
152+
} finally {
153+
if (!successorClosed) {
154+
successor.close();
155+
}
156+
}
157+
} finally {
158+
Unsafe.free(buf, 16, MemoryTag.NATIVE_DEFAULT);
159+
manager.close();
160+
}
161+
});
162+
}
163+
}

‎core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/CursorSendEngineTest.java‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,35 @@ public void testAcknowledgePropagatesToRing() throws Exception {
105105
});
106106
}
107107

108+
@Test
109+
public void testCheckDurabilitySurfacesLatchedFailureToSenderEntryPoints() throws Exception {
110+
TestUtils.assertMemoryLeak(() -> {
111+
// QwpWebSocketSender.flushAndGetSequence()/awaitAckedFsn() call
112+
// engine.checkDurability() as their FIRST durability act, so this
113+
// delegation seam is what stands between a latched periodic
114+
// data-barrier failure and a producer that keeps publishing into
115+
// an unsyncable slot. It had zero test references before this
116+
// pin. Contract: quiet when clean, throws the LATCHED instance
117+
// (not a copy) on every call until the manager's healed pass
118+
// clears it -- callers poll it, so it must be repeatable, not
119+
// one-shot.
120+
try (CursorSendEngine engine = new CursorSendEngine(tmpDir, 4096)) {
121+
engine.checkDurability(); // clean: must not throw
122+
MmapSegmentException failure = new MmapSegmentException("injected data-sync failure");
123+
engine.getRingForTesting().recordDurabilityFailureForTesting(failure);
124+
for (int i = 0; i < 2; i++) {
125+
try {
126+
engine.checkDurability();
127+
fail("latched durability failure must surface on call #" + i);
128+
} catch (MmapSegmentException expected) {
129+
assertTrue("the latched instance itself must surface",
130+
expected == failure);
131+
}
132+
}
133+
}
134+
});
135+
}
136+
108137
@Test
109138
public void testAppendChecksLatchedDurabilityFailureBeforePublishing() throws Exception {
110139
TestUtils.assertMemoryLeak(() -> {

0 commit comments

Comments
 (0)