RRC: fix DC crasher on interleaved leg handovers (orphaned NR-leg entities)

Root cause of the long-standing test_numerology MultiCell-CBR-DL crasher
("second establishment path during interleaved leg handovers" noted in
361b471; also failed under v1.4.3-1.4.5, never baselined). Traced with
per-UE event logging: ue[4] (LTE 1029 / NR 2053), crash at t=10.05s.

Two defects, both on the teardown side:

1. HandoverController: orphaned NR-leg entities at the DC secondary.
   Data-driven flow establishment (Binder::establishUnidirectionalDataConnection)
   provisions NR-leg entities (MAC connection, RLC entities, bypass PDCP) for
   the UE at the master's secondary gNB regardless of whether the UE's NR leg
   is attached to it. If the LTE leg then hands over while the NR leg is
   detached, deleteOldBuffers() cleaned only the master -- the secondary's
   per-UE state was orphaned (t=5.075 provisioning, t=6.05 orphaning in the
   trace). When the UE later returned to that master, re-establishment of the
   same DRBs collided with the leftovers: duplicate-CID ASSERT in
   LteMacBase::createOutgoingConnection (debug) / duplicate rlc-um-tx module
   error (release) at t=10.05s. Fix: deleteOldBuffers() now also cleans the
   old serving node's secondary (via the NR leg's controller) when the NR leg
   is detached; the attached case was already handled by the forced-detachment
   path. Also pass the servingNodeId argument through to the UE-side
   deleteQueues()/deleteLocalPdcpEntities() calls instead of the member
   (identical for all pre-existing call sites, required for the new cross-leg
   call where the NR leg's member is NODEID_NONE).

2. BearerManagement::deleteLocalPdcpEntities() wiped ALL PDCP entities at any
   UE, but at an NR UE this single module holds the entities of BOTH legs
   (keyed by peer node). An NR-leg detach thus also deleted the LTE leg's PDCP
   RX entities, leaving the LTE RLC RX entities forwarding to a dangling gate:
   "Gate 'out' not connected" in UmRxEntity at t=15.03s (exposed only after
   fix 1 let the run get past 10.05s). Fix: keyed (per-node) deletion at eNBs
   and NR-capable UEs, wipe-all only at plain LTE UEs -- exactly the
   pre-flattening per-class semantics (LtePdcpEnb/NrPdcpUe keyed with the
   "delete connections related to the given master nodeB only" comment,
   LtePdcpUe wipe-all).

Verified: test_numerology MultiCell-CBR-DL and MultiCell-CBR-UL now run the
full 50s in both debug and release with zero errors.

Fingerprints: full suite re-run has zero errors and exactly two justified
moves, re-recorded here:
- test_numerology/MultiCell-CBR-DL: interval extended 5s -> 20s (deferred from
  1886a06, blocked on this crasher); all ingredients move with the longer
  window, which now covers the interleaved DC handovers.
- dualConnectivity_multicell/SplitBearer-CBR-DL: tplx/tilx/sz moved, ~tNl
  byte-identical -- the keyed PDCP cleanup changes the entity-deletion event
  trajectory around handovers, while the application packet stream is
  unchanged. All other rows (incl. all other DC, handover, D2D, and MEC
  configs) byte-identical.

Author: avarga

src/simu5g/stack/rrc/BearerManagement.cc

@@ -390,9 +390,17 @@ void BearerManagement::deleteLocalPdcpEntities(MacNodeId nodeId)
 
     bool isEnb = (registration_->getNodeType() == NODEB);
 
+    // Per-node (keyed) deletion at eNBs/gNBs and at NR-capable UEs; wipe-all only at plain
+    // LTE UEs. This mirrors the pre-flattening per-class behavior (LtePdcpEnb and NrPdcpUe
+    // deleted entities keyed by node, LtePdcpUe deleted all). At an NR UE this single module
+    // holds the PDCP entities of BOTH legs (keyed by the peer node): a one-leg detach must
+    // not delete the other leg's entities, otherwise that leg's RLC RX entities are left
+    // forwarding to a dangling gate, and later re-establishment collides with its leftovers.
+    bool keyed = isEnb || registration_->getNrNodeId() != NODEID_NONE;
+
     // Delete PDCP TX entities
     for (auto it = pdcpTxEntities_.begin(); it != pdcpTxEntities_.end(); ) {
-        if (isEnb ? it->first.getNodeId() == nodeId : true) {
+        if (!keyed || it->first.getNodeId() == nodeId) {
             pdcpMux->unregisterTxEntity(it->first);
             it->second->deleteModule();
             it = pdcpTxEntities_.erase(it);
@@ -401,7 +409,7 @@ void BearerManagement::deleteLocalPdcpEntities(MacNodeId nodeId)
 
     // Delete PDCP RX entities
     for (auto it = pdcpRxEntities_.begin(); it != pdcpRxEntities_.end(); ) {
-        if (isEnb ? it->first.getNodeId() == nodeId : true) {
+        if (!keyed || it->first.getNodeId() == nodeId) {
             it->second->deleteModule();
             it = pdcpRxEntities_.erase(it);
         } else ++it;
@@ -410,7 +418,7 @@ void BearerManagement::deleteLocalPdcpEntities(MacNodeId nodeId)
     // Delete bypass TX entities (eNB-only)
     ASSERT(pdcpBypassTxEntities_.empty() || pdcpDcMux != nullptr);
     for (auto it = pdcpBypassTxEntities_.begin(); it != pdcpBypassTxEntities_.end(); ) {
-        if (isEnb ? it->first.getNodeId() == nodeId : true) {
+        if (!keyed || it->first.getNodeId() == nodeId) {
             pdcpDcMux->unregisterBypassTxEntity(it->first);
             it->second->deleteModule();
             it = pdcpBypassTxEntities_.erase(it);
@@ -419,7 +427,7 @@ void BearerManagement::deleteLocalPdcpEntities(MacNodeId nodeId)
 
     // Delete bypass RX entities
     for (auto it = pdcpBypassRxEntities_.begin(); it != pdcpBypassRxEntities_.end(); ) {
-        if (isEnb ? it->first.getNodeId() == nodeId : true) {
+        if (!keyed || it->first.getNodeId() == nodeId) {
             it->second->deleteModule();
             it = pdcpBypassRxEntities_.erase(it);
         } else ++it;

src/simu5g/stack/rrc/HandoverController.cc

@@ -490,7 +490,7 @@ void HandoverController::deleteOldBuffers(MacNodeId servingNodeId)
     servingNodeMac->deleteQueues(nodeId_);
 
     // delete queues for serving node at this UE
-    mac_->deleteQueues(servingNodeId_);
+    mac_->deleteQueues(servingNodeId);
 
     // Delete RLC UM Buffers
 
@@ -515,7 +515,19 @@ void HandoverController::deleteOldBuffers(MacNodeId servingNodeId)
         servingBm->deleteLocalPdcpEntities(nodeId_);
 
     // delete PDCP entities for serving node at this UE
-    bearerManagement_->deleteLocalPdcpEntities(servingNodeId_);
+    bearerManagement_->deleteLocalPdcpEntities(servingNodeId);
+
+    // Flow establishment (Binder::establishUnidirectionalDataConnection) provisions NR-leg
+    // entities for this UE at the serving node's DC secondary regardless of whether the UE's
+    // NR leg is attached to it. If the NR leg is attached, its own (forced) detachment cleans
+    // them up; if it is detached now, remove them together with the master-side state --
+    // otherwise they are orphaned here, and a later re-establishment collides with them when
+    // the UE returns to this master (duplicate MAC CID assert / duplicate module errors).
+    if (otherHandoverController_ != nullptr && otherHandoverController_->getServingNodeId() == NODEID_NONE) {
+        MacNodeId secondaryNodeId = binder_->getSecondaryNode(servingNodeId);
+        if (secondaryNodeId != NODEID_NONE)
+            otherHandoverController_->deleteOldBuffers(secondaryNodeId);
+    }
 }
 
 LteAmc *HandoverController::getAmcModule(MacNodeId nodeId)

tests/fingerprint/simulations.csv

@@ -72,7 +72,7 @@
 /simulations/nr/dualConnectivity_multicell/, -f omnetpp.ini -c DoubleConnection-CBR-UL -r 0,    40s,         05c6-61aa/tplx;d0f2-6422/tilx;3944-e298/~tNl;3c35-155e/sz, PASS,
 /simulations/nr/dualConnectivity_multicell/, -f omnetpp.ini -c SingleConnection-CBR-DL -r 0,    40s,         f85c-f458/tplx;9b59-7b8b/tilx;44af-6cd8/~tNl;34d9-dab2/sz, PASS,
 /simulations/nr/dualConnectivity_multicell/, -f omnetpp.ini -c SingleConnection-CBR-UL -r 0,    40s,         1e43-8b02/tplx;c59a-41a0/tilx;39c2-5c6b/~tNl;c0f8-dd1a/sz, PASS,
-/simulations/nr/dualConnectivity_multicell/, -f omnetpp.ini -c SplitBearer-CBR-DL -r 0,         40s,         b45c-c106/tplx;14cc-a55a/tilx;e332-82db/~tNl;33e0-ab8f/sz, PASS,
+/simulations/nr/dualConnectivity_multicell/, -f omnetpp.ini -c SplitBearer-CBR-DL -r 0,         40s,         daf2-a2d8/tplx;bea5-8ebb/tilx;e332-82db/~tNl;4765-2a49/sz, PASS,
 /simulations/nr/dualConnectivity/,       -f omnetpp.ini -c DualConn-MasterOnly-DL -r 0,          5s,         27da-87eb/tplx;dbfe-88dd/tilx;7596-fd84/~tNl;3cbd-e6df/sz, PASS,
 /simulations/nr/dualConnectivity/,       -f omnetpp.ini -c DualConn-MasterOnly-UL -r 0,          5s,         6c5c-3dd7/tplx;ce67-3b2e/tilx;02a2-0a09/~tNl;575a-4aaf/sz, PASS,
 /simulations/nr/dualConnectivity/,       -f omnetpp.ini -c DualConn-MasterPlusSecondary-DL -r 0, 5s,         1544-1cc7/tplx;a006-c15a/tilx;ea4a-5e6e/~tNl;bbf0-0344/sz, PASS,
@@ -115,7 +115,7 @@
 /simulations/nr/test_multiCarrier/,      -f omnetpp.ini -c MultiCarrier-CBR-UL -r 0,             5s,         2cdd-48da/tplx;bfa7-1cf0/tilx;8d1b-cbd7/~tNl;c822-5817/sz, PASS,
 /simulations/nr/test_multiCarrier/,      -f omnetpp.ini -c SingleCarrier-CBR-DL -r 0,            5s,         3439-c327/tplx;d254-3b41/tilx;87df-bb05/~tNl;836c-a2bd/sz, PASS,
 /simulations/nr/test_multiCarrier/,      -f omnetpp.ini -c SingleCarrier-CBR-UL -r 0,            5s,         4e36-f5e0/tplx;e3af-85cb/tilx;1400-0221/~tNl;5a3f-ae80/sz, PASS,
-/simulations/nr/test_numerology/,        -f omnetpp.ini -c MultiCell-CBR-DL -r 0,                5s,         b433-bcd7/tplx;1ae8-9e78/tilx;6284-1d6c/~tNl;d4eb-f413/sz, PASS,
+/simulations/nr/test_numerology/,        -f omnetpp.ini -c MultiCell-CBR-DL -r 0,               20s,         5b20-53f9/tplx;eaad-b1a5/tilx;def6-7c78/~tNl;e0cc-0380/sz, PASS,
 /simulations/nr/test_numerology/,        -f omnetpp.ini -c MultiCell-CBR-UL -r 0,               20s,         f09a-c9bb/tplx;a8c5-5c41/tilx;1221-b278/~tNl;627c-015e/sz, PASS,
 /simulations/nr/test_numerology/,        -f omnetpp.ini -c SingleCell-CBR-DL -r 0,              10s,         7168-a79d/tplx;d8f5-b1b2/tilx;662f-87d8/~tNl;718b-31ac/sz, PASS,
 /simulations/nr/test_numerology/,        -f omnetpp.ini -c SingleCell-CBR-UL -r 0,              10s,         9a96-9c9d/tplx;2f0b-ad1d/tilx;cf96-908e/~tNl;e1be-7696/sz, PASS,