Skip to content

fix: prevent restartable outbox relay cancellation from faulting the host on .NET 10#310

Merged
Vulthil merged 1 commit into
mainfrom
fix/outbox-relay-restart-race
Jul 9, 2026
Merged

fix: prevent restartable outbox relay cancellation from faulting the host on .NET 10#310
Vulthil merged 1 commit into
mainfrom
fix/outbox-relay-restart-race

Conversation

@Vulthil

@Vulthil Vulthil commented Jul 9, 2026

Copy link
Copy Markdown
Owner

Summary

WebApi.Tests.ExternalWeatherClientTests intermittently failed with ObjectDisposedException on the shared factory's IServiceProvider after the host logged Host[9] BackgroundService failed: TaskCanceledException with no user frames (latest: CI run 29039323993 attempt 1). Root cause: on .NET 10, BackgroundService.StartAsync schedules ExecuteAsync via Task.Run(..., stoppingToken). When the test harness's per-test reset stops the restartable outbox relay before the thread pool has dequeued that delegate, the host-observed first execute task transitions straight to canceled without ExecuteAsync ever running — and because the application is not shutting down, the default BackgroundServiceExceptionBehavior.StopHost stops the host mid-class. No in-method cancellation handling can prevent this, which is why the earlier relay-gate and consumer-retry hardening did not close the flake.

  • OutboxBackgroundService no longer inherits BackgroundService; it implements IRestartableHostedService + IDisposable and owns its lifecycle: the execute task is scheduled with an ungated Task.Run, cancellation is observed inside the loop and always ends in a graceful return, the host never awaits the task, and a genuine fault still stops the application via IHostApplicationLifetime.StopApplication() (log: critical), matching the BackgroundService default.
  • OutboxRetentionBackgroundService / InboxRetentionBackgroundService: a foreign OperationCanceledException thrown by the store while the application is running (for example a client library surfacing a timeout as a cancellation) is now logged and retried on the next tick instead of escaping and faulting the host; graceful shutdown ends the execute task in RanToCompletion.
  • IRestartableHostedService XML docs and the Vulthil.Extensions.Hosting README now recommend implementing IHostedService directly for restartable services and explain the .NET 10 hazard.

No public API changes (all touched service types are internal; the interface change is documentation only).

Verification

  • New regression tests: StartingWithAnAlreadyCanceledTokenCompletesTheExecuteTaskGracefully deterministically reproduces the pre-execution cancellation endpoint; StoppingBeforeTheRelayLoopHasRunCompletesGracefully mirrors the harness stop sequence; restart-cycle and fault-stops-application tests pin the new lifecycle; both retention suites assert a foreign cancellation does not end the execute task. Mutation-checked: reintroducing the token-gated Task.Run and the old sweep filter makes three of them fail.
  • Full solution builds clean; Vulthil.SharedKernel.Outbox.Tests and Vulthil.Messaging.Inbox.Tests pass on net9.0 and net10.0.
  • WebApi.Tests passes 8/8 end-to-end against PostgreSQL + RabbitMQ containers, exercising the per-test relay restart.

Backport to v1.0: yes

@Vulthil Vulthil added the backport v1.0 Backport this PR to the v1.0 servicing branch label Jul 9, 2026
@Vulthil
Vulthil merged commit 041f40d into main Jul 9, 2026
7 checks passed
@Vulthil
Vulthil deleted the fix/outbox-relay-restart-race branch July 9, 2026 19:49
@Vulthil

Vulthil commented Jul 9, 2026

Copy link
Copy Markdown
Owner Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport v1.0 Backport this PR to the v1.0 servicing branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant