Symptom
InitOnLineTest failed on CI (ubuntu-latest/11 leg of PR #714's run, unrelated
to that PR) — 3 tests, all with the same assertion in per-test cleanup:
InitOnLineTest.initializeTargetExportMultiSS:1026->afterTest:1312
InitOnLineTest.initializeTargetImport:786->afterTest:1312
InitOnLineTest.testReplServerInfos:970->afterTest:1312
ReplicationDomain: Import/Export is not expected to be running expected [false] but found [true]
The three failing tests are the last three in execution order — a cascade
started by the first one. The server log pinpoints the trigger inside
initializeTargetExportMultiSS:
category=BACKEND severity=ERROR msgID=99 msg=An error occurred while executing the task
ds-task-id=...: DirectoryException: Cannot start total update in domain "dc=example,dc=com"
from this directory server DS(1): the remote directory server DS(2) is unknown
(ReplicationDomain.java:1482 LDAPReplicationDomain.java:1521 ReplicationDomain.java:1421
InitializeTargetTask.java:99 Task.java:965 TaskThread.java:179)
The test connected broker DS(2) and immediately scheduled the
InitializeTarget task; on the loaded runner the task started before DS(1)
had learned about DS(2) from the topology — a benign, expected-to-be-retryable
startup error. What is not benign is what it left behind.
Bug 1 (product): failed pre-export validation leaks the import/export context
ReplicationDomain.initializeRemote() acquires the import/export context
before validating the request, and the validation errors escape without
releasing it:
final ImportExportContext ieCtx = acquireIEContext(false); // line ~1445
...
if (replicaInfos.isEmpty())
throw new DirectoryException(..., ERR_FULL_UPDATE_NO_REMOTES...); // leaks ieCtx
...
if (dsi == null)
throw new DirectoryException(..., ERR_FULL_UPDATE_MISSING_REMOTE...); // leaks ieCtx <- this CI failure
The only release is far below and not exception-safe — the code even carries
an upstream FIXME acknowledging it:
releaseIEContext(); // FIXME should not this be in a finally? // line ~1629
Once leaked, ieRunning() stays true forever, and every subsequent
initialize attempt on that domain fails with
ERR_SIMULTANEOUS_IMPORT_EXPORT_REJECTED (the compareAndSet in
acquireIEContext can never succeed again).
Production impact: a single dsreplication initialize pointed at a
not-yet-connected / unknown server id permanently poisons the domain — all
later total updates are rejected as "simultaneous import/export" until the
server is restarted.
Fix direction: move validation before acquireIEContext(), and/or put
releaseIEContext() in a finally as the FIXME already suggests.
Bug 2 (test): afterTest asserts before cleaning up, turning one flake into a cascade
InitOnLineTest.afterTest() first waits up to 5 s for ieRunning() to clear
and asserts before any cleanup:
assertFalse(replDomain.ieRunning(), "..."); // line 1312 - throws here...
super.cleanConfigEntries(); // ...so none of this runs:
stop(server2, server3); // brokers leak
remove(replServer1, replServer2, replServer3); // three RSs leak
When bug 1 strikes, the assertion throws, the whole environment (domain
config, 2 brokers, 3 replication servers) leaks into the following tests, and
they fail the same way — hence 3 failures from one root event.
Fix direction: run the cleanup in a finally (or move the assertion after
cleanup), and wait for DS(1) to see the target server id in
getReplicaInfos() before scheduling InitializeTarget tasks.
References
Symptom
InitOnLineTestfailed on CI (ubuntu-latest/11 leg of PR #714's run, unrelatedto that PR) — 3 tests, all with the same assertion in per-test cleanup:
The three failing tests are the last three in execution order — a cascade
started by the first one. The server log pinpoints the trigger inside
initializeTargetExportMultiSS:The test connected broker DS(2) and immediately scheduled the
InitializeTargettask; on the loaded runner the task started before DS(1)had learned about DS(2) from the topology — a benign, expected-to-be-retryable
startup error. What is not benign is what it left behind.
Bug 1 (product): failed pre-export validation leaks the import/export context
ReplicationDomain.initializeRemote()acquires the import/export contextbefore validating the request, and the validation errors escape without
releasing it:
The only release is far below and not exception-safe — the code even carries
an upstream FIXME acknowledging it:
Once leaked,
ieRunning()staystrueforever, and every subsequentinitialize attempt on that domain fails with
ERR_SIMULTANEOUS_IMPORT_EXPORT_REJECTED(thecompareAndSetinacquireIEContextcan never succeed again).Production impact: a single
dsreplication initializepointed at anot-yet-connected / unknown server id permanently poisons the domain — all
later total updates are rejected as "simultaneous import/export" until the
server is restarted.
Fix direction: move validation before
acquireIEContext(), and/or putreleaseIEContext()in afinallyas the FIXME already suggests.Bug 2 (test): afterTest asserts before cleaning up, turning one flake into a cascade
InitOnLineTest.afterTest()first waits up to 5 s forieRunning()to clearand asserts before any cleanup:
When bug 1 strikes, the assertion throws, the whole environment (domain
config, 2 brokers, 3 replication servers) leaks into the following tests, and
they fail the same way — hence 3 failures from one root event.
Fix direction: run the cleanup in a finally (or move the assertion after
cleanup), and wait for DS(1) to see the target server id in
getReplicaInfos()before scheduling InitializeTarget tasks.References