Skip to content

initializeRemote() leaks import/export context on failed validation: domain permanently rejects total updates (InitOnLineTest CI cascade) #730

Description

@vharseko

Symptom

InitOnLineTest failed on CI (ubuntu-latest/11 leg of PR #714's run, unrelated
to that PR) — 3 tests, all with the same assertion in per-test cleanup:

InitOnLineTest.initializeTargetExportMultiSS:1026->afterTest:1312
InitOnLineTest.initializeTargetImport:786->afterTest:1312
InitOnLineTest.testReplServerInfos:970->afterTest:1312
  ReplicationDomain: Import/Export is not expected to be running expected [false] but found [true]

The three failing tests are the last three in execution order — a cascade
started by the first one. The server log pinpoints the trigger inside
initializeTargetExportMultiSS:

category=BACKEND severity=ERROR msgID=99 msg=An error occurred while executing the task
ds-task-id=...: DirectoryException: Cannot start total update in domain "dc=example,dc=com"
from this directory server DS(1): the remote directory server DS(2) is unknown
(ReplicationDomain.java:1482 LDAPReplicationDomain.java:1521 ReplicationDomain.java:1421
 InitializeTargetTask.java:99 Task.java:965 TaskThread.java:179)

The test connected broker DS(2) and immediately scheduled the
InitializeTarget task; on the loaded runner the task started before DS(1)
had learned about DS(2) from the topology — a benign, expected-to-be-retryable
startup error. What is not benign is what it left behind.

Bug 1 (product): failed pre-export validation leaks the import/export context

ReplicationDomain.initializeRemote() acquires the import/export context
before validating the request, and the validation errors escape without
releasing it:

final ImportExportContext ieCtx = acquireIEContext(false);   // line ~1445
...
if (replicaInfos.isEmpty())
  throw new DirectoryException(..., ERR_FULL_UPDATE_NO_REMOTES...);      // leaks ieCtx
...
if (dsi == null)
  throw new DirectoryException(..., ERR_FULL_UPDATE_MISSING_REMOTE...);  // leaks ieCtx  <- this CI failure

The only release is far below and not exception-safe — the code even carries
an upstream FIXME acknowledging it:

releaseIEContext(); // FIXME should not this be in a finally?            // line ~1629

Once leaked, ieRunning() stays true forever, and every subsequent
initialize attempt on that domain fails with
ERR_SIMULTANEOUS_IMPORT_EXPORT_REJECTED (the compareAndSet in
acquireIEContext can never succeed again).

Production impact: a single dsreplication initialize pointed at a
not-yet-connected / unknown server id permanently poisons the domain — all
later total updates are rejected as "simultaneous import/export" until the
server is restarted.

Fix direction: move validation before acquireIEContext(), and/or put
releaseIEContext() in a finally as the FIXME already suggests.

Bug 2 (test): afterTest asserts before cleaning up, turning one flake into a cascade

InitOnLineTest.afterTest() first waits up to 5 s for ieRunning() to clear
and asserts before any cleanup:

assertFalse(replDomain.ieRunning(), "...");   // line 1312 - throws here...
super.cleanConfigEntries();                    // ...so none of this runs:
stop(server2, server3);                        // brokers leak
remove(replServer1, replServer2, replServer3); // three RSs leak

When bug 1 strikes, the assertion throws, the whole environment (domain
config, 2 brokers, 3 replication servers) leaks into the following tests, and
they fail the same way — hence 3 failures from one root event.

Fix direction: run the cleanup in a finally (or move the assertion after
cleanup), and wait for DS(1) to see the target server id in
getReplicaInfos() before scheduling InitializeTarget tasks.

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugreplicationtestsTest suites: fixing, enabling, un-disabling

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions