Version: 5.1.1, also present on master.

Summary

When a client dereferences aliases, SearchOperationBasis.returnEntry records the DN of every entry it returns in a set that lives as long as the search operation. Nothing ever bounds or clears it:

opendj-server-legacy/src/main/java/org/opends/server/core/SearchOperationBasis.java#L452

Set<DN> dereferenced=new HashSet<>();

opendj-server-legacy/src/main/java/org/opends/server/core/SearchOperationBasis.java#L566-L585

//DereferenceAliasesPolicy
if ( DereferenceAliasesPolicy.ALWAYS.equals(getDerefPolicy()) || DereferenceAliasesPolicy.IN_SEARCHING.equals(getDerefPolicy()) ) {
  if (entry.isAlias() && !baseDN.equals(entry.getName())) {
    try {
      final DN dn = entry.getAliasedDN();
      final Entry dereference = DirectoryServer.getEntry(dn);
      if (dereferenced.contains(dn)) {
        return true;
      }
      dereferenced.add(dn);
      return returnEntry(dereference, controls, true);
    } catch (DirectoryException e) {
      throw new RuntimeException(e);
    }
  }
  if (dereferenced.contains(entry.getName())) {
    return true;
  }
  dereferenced.add(entry.getName());
}

The set is only meant to stop an alias loop from returning the same target twice, but the dereferenced.add(entry.getName()) on the last line runs for every ordinary, non-alias entry as well.

Impact

Persistent searches grow without bound. PersistentSearch keeps its SearchOperation for the lifetime of the search (PersistentSearch.java#L128) and calls returnEntry on it for every change it notifies (PersistentSearch.java#L391). LocalBackendSearchOperation is a SearchOperationWrapper and delegates straight through to the single SearchOperationBasis instance, so a long-running persistent search with derefAliases: always accumulates one DN per notified entry, forever, until the server runs out of memory.

This is reachable by any authenticated client allowed to run a persistent search, and JNDI-based clients send derefAliases: always unless told otherwise. It is the same shape as CVE-2025-27497 (alias loop DoS, fixed in 08aee47), but on the result path rather than the base DN path — 08aee47 does not touch this code.

Ordinary searches pay an avoidable O(N) allocation. For a non-persistent search the set is freed when the operation ends, so it is not a leak, but a large result set (a full cn=changelog dump, a wide subtree search) now retains one DN per returned entry for the duration of the search, for no benefit on entries that are not aliases.

Note this path becomes reachable for cn=changelog once #737 is fixed: before that fix, such searches fail earlier with result code 80.

Adjacent defects in the same block

Dangling alias causes an NPE → result code 80. DirectoryServer.getEntry(dn) returns null when the alias target does not exist, and the result is passed straight into the recursive returnEntry(dereference, controls, true) on line 576 without a null check. The recursion then dereferences it at entry.isSubentry():

opendj-server-legacy/src/main/java/org/opends/server/core/SearchOperationBasis.java#L481

if (entry.isSubentry() || entry.isLDAPSubentry())

The NPE unwinds into the backend search loop and is caught by the generic catch (Exception e) in LocalBackendSearchOperation, so the client gets error code 80 instead of the entry simply being skipped. Aliases are not referentially checked on write, so a dangling alias is easy to create.

DirectoryException is wrapped in a RuntimeException on line 578, which likewise degrades a proper result code into an error 80.

Steps to reproduce

For the unbounded growth, run a persistent search with alias dereferencing against a busy suffix and watch the heap:

ldapsearch -h localhost -p 1389 -D "cn=Directory Manager" -w password \
    -b "o=test" -s sub -a always -C ps:any:1:1 "(objectclass=*)"

Each notified change adds a DN to the operation's dereferenced set, which is never trimmed.

For the dangling alias NPE:

dn: ou=broken,o=test
objectClass: top
objectClass: alias
objectClass: extensibleObject
ou: broken
aliasedObjectName: ou=does-not-exist,o=test

# error 80 instead of skipping the alias
ldapsearch -h localhost -p 1389 -D "cn=Directory Manager" -w password \
    -b "o=test" -s sub -a always "(objectclass=*)"

Regression

Introduced with the alias dereferencing feature in f771315 ("[#287] ADD alias dereferencing for search requests", #369, 2024-08-07).

Suggested fix

• Only track DNs actually reached by dereferencing an alias, not every returned entry, so the set stays proportional to the number of aliases rather than to the result set.
• Do not accumulate across the persistent search phase — the loop guard is only meaningful within one traversal, so the state should not outlive it.
• Skip the entry when the alias target resolves to null instead of recursing into it.
• Let DirectoryException propagate as such rather than wrapping it in a RuntimeException.

Note

Found while analysing #737. See also #738, a third defect in the same feature.