Fix nightly Stress Tests (Chaos TCP): transparent reconnect + robust chaos assertions (#3986)

# Description

Fixes the nightly **Stress Tests / Chaos TCP** job (e.g. run
[29068846476](https://github.com/OPCFoundation/UA-.NETStandard/actions/runs/29068846476)),
which had failed on **every run since it was introduced** — the workflow
runs on a schedule and never gated PRs, so the failures went unnoticed.

## Root cause

The Chaos TCP tests point a `ConfiguredEndpoint` at a local
`TcpChaosProxy` and set `UpdateBeforeConnect = false` so the session
connects *through* the proxy. Two problems prevented this from working:

1. `ManagedSession.HandleConnectAsync` **ignored**
`ConfiguredEndpoint.UpdateBeforeConnect` and hard-coded a pre-connect
re-discovery for all non-OpenAPI endpoints. Re-discovery adopted the
server's advertised `EndpointUrl`, so the session connected **directly
to the server, bypassing the proxy** — no chaos ever reached the
connection. Two tests "passed" vacuously.
2. Once the proxy was genuinely engaged, the transparent-reconnect
feature had a gap: a request that hit the just-dropped channel (before
the manager detected the drop) failed with `BadConnectionClosed` instead
of transparently recovering.

## Changes

**Product**

- **`ManagedSession`** now honours
`ConfiguredEndpoint.UpdateBeforeConnect` (default `true`). Setting it
`false` opens the channel against exactly the supplied URL (proxy /
gateway / NAT / pinned endpoint) without re-discovering.
- **`ManagedTransportChannelLease.SendRequestAsync`** transparently
resends **idempotent** requests (`Read` / `Browse` /
`TranslateBrowsePaths` / `GetEndpoints` / `FindServers`) across a
channel-manager reconnect. On a transient transport-drop error it forces
a *coalesced* reconnect and resends once the shared channel recovers.
- A new **`ChannelEntry.ReconnectGeneration`** counter guards this so a
reconnect is only forced for a genuinely undetected drop (same
generation, still `Ready`), never for a stale in-flight failure that
arrives after recovery — which previously spawned spurious extra
reconnect cycles that tore down healthy channels.
- **Non-idempotent** requests (`Create*`, `Write`, `Call`, `Publish`, …)
surface the error so higher-level recovery (e.g. the subscription
engine's own re-create loop) stays in control; requests are never
double-applied.

**Tests / harness**

- **`TcpChaosProxy`** no longer rethrows the expected `OperationAborted`
raised by an in-flight upstream `ConnectAsync` during
`DropAllConnections` (it was faulting the awaited connection task and
failing the test).
- Chaos reconnect assertions (L3-A1/A2/A5) are made robust to the
inherent non-determinism of reconnect **counts** under concurrent chaos
(coalescing merges drops landing in one recovery window; a stale
keep-alive `Bad` can add one cycle). Survival is asserted via
`FailureRate`, `ReconnectFailed == 0`, and the coalescing fan-out
relationship rather than exact `ReconnectStarted/Completed ==
dropCount`.
- Subscription post-drop recovery window widened (`2s → 8s`) to allow
re-creating all sessions' subscriptions and monitored items on a loaded
CI agent.
- **New unit tests** in `ClientChannelManagerManagedTests` cover the new
product paths (fast-PR / codecov, since the nightly ChaosTCP job does
not run in the fast pipeline): idempotent transient-drop retry,
non-idempotent no-retry, non-transient no-retry, and
`ChannelEntry.ReconnectGeneration` increment.

## Validation (local, net10.0)

- **Chaos TCP**: 6/6 pass across many random seeds (incl. post-merge
re-runs of `338475986` and `1999999999`).
- `Opc.Ua.Client.Tests` ChannelManager/ManagedSession: **256/256**
(incl. the 4 new tests).
- `Opc.Ua.Sessions.Tests` ChannelManager/Reconnect: **139/139**.
- Core / Client / Stress build with **0 warnings**.

## Notes

- This branch has been merged up to date with `master` (redundancy PR
#3918).
- The two failing `Opc.Ua.PubSub.Tests` in CI —
`UdpLoopbackActionResponderAnswersRequesterAsync` and
`UdpLoopbackDiscoveryPublisherAnswersSubscriberRequests` — are
**unrelated** to this PR (UDP-loopback request/response tests in the
PubSub library, associated with #3918) and are timing/environment-flaky.
They are out of scope for this change.

## Related Issues

- Fixes the recurring nightly Chaos TCP failure (workflow:
`.github/workflows/stress-test.yml`).

## Checklist

- [ ] I have signed the
[CLA](https://opcfoundation.org/license/cla/ContributorLicenseAgreementv1.0.pdf)
and read the
[CONTRIBUTING](https://github.com/OPCFoundation/UA-.NETStandard/blob/master/CONTRIBUTING.md)
doc.
- [x] I have added tests that prove my fix is effective or that my
feature works and increased code coverage.
- [ ] I have added all necessary documentation.
- [x] I have verified that my changes do not introduce (new) build or
analyzer warnings.
- [ ] I ran **all** tests locally using the **UA.slnx** solution against
at least .net **framework** and .net **10**, and all passed.
- [ ] I fixed **all** failing and flaky tests in the CI pipelines and
**all** CodeQL warnings.
- [ ] I have addressed **all** PR feedback received.

Author: marcschier

.github/workflows/preview-publish.yml

@@ -191,7 +191,7 @@ jobs:
         # Scoped to nuget/ - the MigrationAnalyzer nuspec at Tools/Opc.Ua.MigrationAnalyzer/
         # uses MSBuild $(...) substitutions that only `dotnet pack` resolves; standalone
         # `nuget.exe pack` would crash on it. See PR #3872.
-        continueOnError: true
+        continue-on-error: true
         run: |
           $nuspecs = Get-ChildItem -Path 'nuget' -Filter 'Opc.*.nuspec' -File
           if (-not $nuspecs) {
@@ -209,7 +209,7 @@ jobs:
       - name: Sign NuGet packages (nupkg + snupkg)
         if: env.SIGNING_CLIENT_SECRET != ''
         shell: pwsh
-        continueOnError: true
+        continue-on-error: true
         run: |
           $nupkgs  = Get-ChildItem -Path "$env:ARTIFACTS_DIR" -Filter 'OPCFoundation.*.nupkg'  -File
           $snupkgs = Get-ChildItem -Path "$env:ARTIFACTS_DIR" -Filter 'OPCFoundation.*.snupkg' -File

Libraries/Opc.Ua.Client/Session/ManagedSession.cs

@@ -990,9 +990,17 @@ private async Task<ServiceResult> HandleConnectAsync(
                 // channel manager would fail to resolve a factory for
                 // the bare "https" scheme. Skip the refresh for the
                 // OpenAPI profiles to preserve the caller-supplied URL.
+                //
+                // Otherwise honour the caller's ConfiguredEndpoint.UpdateBeforeConnect
+                // choice (default true). A caller that pins a specific URL - for
+                // example connecting through a proxy / gateway / NAT whose host and
+                // port differ from the server's advertised EndpointUrl - sets the
+                // flag to false so the channel is opened against exactly that URL
+                // instead of re-discovering and adopting the server-advertised URL.
                 string? profile = ConfiguredEndpoint.Description.TransportProfileUri;
-                bool updateBeforeConnect = !Profiles.IsHttpsOpenApi(profile) &&
-                    !Profiles.IsWssOpenApi(profile);
+                bool updateBeforeConnect = ConfiguredEndpoint.UpdateBeforeConnect
+                    && !Profiles.IsHttpsOpenApi(profile)
+                    && !Profiles.IsWssOpenApi(profile);
 
                 IDisposable? connectLease = null;
                 Session session;

Stack/Opc.Ua.Core/Stack/Client/Channels/Internal/ChannelEntry.cs

@@ -99,6 +99,16 @@ public ChannelState State
             }
         }
 
+        /// <summary>
+        /// Monotonic counter incremented each time the entry enters
+        /// <see cref="ChannelState.TransportReconnecting"/>. Callers capture
+        /// it before sending a request and compare afterwards to tell a
+        /// genuinely undetected transport drop (generation unchanged) apart
+        /// from a stale in-flight failure that arrives after the shared
+        /// channel has already begun (or finished) reconnecting.
+        /// </summary>
+        internal long ReconnectGeneration => Interlocked.Read(ref m_reconnectGeneration);
+
         public ITransportChannel? Underlying
         {
             get
@@ -1215,6 +1225,10 @@ private void TransitionTo(ChannelState next, ServiceResult? error, int attempt)
                     return;
                 }
                 m_state = next;
+                if (next == ChannelState.TransportReconnecting)
+                {
+                    Interlocked.Increment(ref m_reconnectGeneration);
+                }
                 m_lastStateChange = OwnerManager.TimeProvider.GetUtcNow();
                 m_lastReconnectAttempt = attempt;
                 m_lastError = error;
@@ -1291,6 +1305,7 @@ private struct AggregatedReactivationOutcome
         private ServiceResult? m_lastError;
         private ChannelState m_state = ChannelState.Disconnected;
         private ITransportChannel? m_underlying;
+        private long m_reconnectGeneration;
         private long m_clientCertificateVersion;
         private TaskCompletionSource<bool> m_readyGate;
         private TaskCompletionSource<bool>? m_reconnectCoalescer;

Stack/Opc.Ua.Core/Stack/Client/Channels/Internal/ManagedTransportChannelLease.cs

@@ -242,13 +242,109 @@ public async ValueTask<IServiceResponse> SendRequestAsync(
                 return await bypass.SendRequestAsync(request, ct).ConfigureAwait(false);
             }
 
-            await Entry.WaitForReadyAsync(ct).ConfigureAwait(false);
+            int attempt = 0;
+            while (true)
+            {
+                await Entry.WaitForReadyAsync(ct).ConfigureAwait(false);
+
+                ChannelEntry entry = Entry;
+                long generation = entry.ReconnectGeneration;
+                ITransportChannel? underlying = entry.Underlying
+                    ?? throw ServiceResultException.Create(
+                        StatusCodes.BadSecureChannelClosed,
+                        "Channel has no underlying transport.");
+                try
+                {
+                    return await underlying.SendRequestAsync(request, ct).ConfigureAwait(false);
+                }
+                catch (ServiceResultException sre) when (
+                    IsActive &&
+                    attempt < kMaxTransientSendRetries &&
+                    !ct.IsCancellationRequested &&
+                    IsIdempotentRequest(request) &&
+                    IsTransientChannelError(sre.StatusCode))
+                {
+                    // The shared channel broke while the request was in
+                    // flight. If no reconnect has begun since we sent (the
+                    // generation is unchanged) and the entry still reports
+                    // Ready, this is a transport drop the manager has not
+                    // detected yet - force a (coalesced) reconnect so the
+                    // shared channel recovers. Otherwise a reconnect is
+                    // already in progress or completed (a stale in-flight
+                    // failure): loop back, wait for the recovered channel and
+                    // resend without starting an extra reconnect cycle. Either
+                    // way the drop is transparent for the caller. Retries are
+                    // bounded and the reconnect itself is governed by the
+                    // reconnect policy / budget; if the channel terminally
+                    // faults the original transport error is surfaced.
+                    attempt++;
+                    ChannelState state = entry.State;
+                    if (state is ChannelState.Closed or ChannelState.Faulted)
+                    {
+                        // The entry is already terminal (e.g. the reconnect
+                        // policy / budget was exhausted by a concurrent cycle).
+                        // Surface the ORIGINAL transport error rather than
+                        // looping back into WaitForReadyAsync, which would throw
+                        // a generic BadSecureChannelClosed and mask the real
+                        // failure signal.
+                        throw;
+                    }
+                    if (entry.ReconnectGeneration == generation &&
+                        state == ChannelState.Ready)
+                    {
+                        bool recovered;
+                        try
+                        {
+                            recovered = await entry.RequestReconnectAsync(ct).ConfigureAwait(false);
+                        }
+                        catch (ServiceResultException)
+                        {
+                            throw sre;
+                        }
+
+                        if (!recovered)
+                        {
+                            throw;
+                        }
+                    }
+                }
+            }
+        }
 
-            ITransportChannel? underlying = Entry.Underlying
-                ?? throw ServiceResultException.Create(
-                    StatusCodes.BadSecureChannelClosed,
-                    "Channel has no underlying transport.");
-            return await underlying.SendRequestAsync(request, ct).ConfigureAwait(false);
+        private static bool IsIdempotentRequest(IServiceRequest request)
+        {
+            // Only requests with no server-side side effects are safe to
+            // transparently re-send after a transport drop: the request may
+            // already have been processed on the server before the
+            // connection died, so re-sending a non-idempotent request
+            // (Create*, Write, Call, AddNodes, HistoryUpdate, Publish,
+            // RegisterNodes, BrowseNext, Query*, ...) could double-apply it
+            // or corrupt continuation state. Non-idempotent requests surface
+            // the transport error instead, letting the caller or a
+            // higher-level recovery loop (for example the subscription
+            // engine's own re-create) decide how to retry.
+            return request is ReadRequest
+                or BrowseRequest
+                or TranslateBrowsePathsToNodeIdsRequest
+                or GetEndpointsRequest
+                or FindServersRequest
+                or FindServersOnNetworkRequest;
+        }
+
+        private static bool IsTransientChannelError(StatusCode statusCode)
+        {
+            uint code = statusCode.CodeBits;
+            return code == StatusCodes.BadConnectionClosed
+                || code == StatusCodes.BadSecureChannelClosed
+                || code == StatusCodes.BadSecureChannelIdInvalid
+                || code == StatusCodes.BadNotConnected
+                || code == StatusCodes.BadConnectionRejected
+                || code == StatusCodes.BadServerNotConnected
+                || code == StatusCodes.BadServerHalted
+                || code == StatusCodes.BadNoCommunication
+                || code == StatusCodes.BadCommunicationError
+                || code == StatusCodes.BadTcpInternalError
+                || code == StatusCodes.BadRequestInterrupted;
         }
 
         /// <inheritdoc/>
@@ -310,6 +406,7 @@ private async ValueTask DisposeAsyncCore()
         private ChannelEntry m_entry;
         private int m_active;
         private int m_swapCount;
+        private const int kMaxTransientSendRetries = 3;
         private readonly Lock m_participantLock = new();
         private IReconnectParticipant m_participant;
     }