Preview deployments can leave orphan Swarm services after PR close

[budivoogt]

Description

Dokploy preview deployments can drift out of sync with Docker Swarm when PR lifecycle events race the preview build/create path.

In our self-hosted Dokploy v0.28.8 setup, we observed two failure modes:

1. Dokploy marks a preview deployment record as running before the Swarm service exists.
2. If the PR is then closed, Dokploy removes the preview deployment record but leaves the Swarm service running, creating an orphan preview service.

This leaves live preview services on the host that are no longer represented in preview_deployments and are therefore not cleaned up by Dokploy.

Environment

• Dokploy v0.28.8
• GitHub integration enabled for preview deployments
• Docker Swarm on a single Hetzner host
• Preview deployments using wildcard subdomains

Controlled reproduction

1. Open a temporary PR against the tracked integration branch.
2. Dokploy authorizes preview creation and inserts a row in preview_deployments with status running.
3. While or after the preview image build completes, inspect Swarm:
   • expected preview service name from Dokploy DB exists in the preview record
   • actual services initially do not contain that service even though the DB row already says running
4. Close the PR without merge.
5. Observe that:
   • the preview_deployments row for that PR disappears from Dokploy DB
   • but the Swarm service appears afterwards and remains running as an orphan

Actual result

Dokploy preview record is deleted, but the preview service remains live in Swarm.

In our controlled test:

• the Dokploy preview record was created while the PR was open
• after closing the PR, the Dokploy DB row was gone
• the corresponding preview service still existed in docker service ls for more than one minute after close

We also found broader drift before cleanup:

• 17 preview Swarm services on the host
• only 3 preview_deployments records in Dokploy DB

So this is not just a one-off transient timing issue.

Expected result

Preview teardown should be idempotent and self-healing:

• if a PR is closed, both the Dokploy preview record and the Swarm service should be removed
• if create/delete events race, Dokploy should reconcile preview DB state against actual Swarm state
• previews should never be left running without a matching Dokploy preview record

Additional notes

We did not find an exact existing Dokploy issue for this specific failure mode. The closest related issue we found was about preview lifecycle/database isolation rather than teardown drift:

• Preview Deployments - Also run database migrations only for each Preview Deployment #2462 Preview Deployments - Also run database migrations only for each Preview Deployment

As an immediate containment step on our side, we added a host-side reconciliation script that compares Dokploy preview_deployments against docker service ls and removes unmanaged preview services. But this should not be necessary if Dokploy teardown is reliable.

[dosubot]

Great bug report — this is a real race condition in the preview deployment lifecycle. Here's what's happening in the code:

Root cause: no synchronization between creation and deletion webhooks.

When a PR is opened, Dokploy creates a DB record, queues a build job, and returns immediately [1]. The worker then sets the status to "running" before the Swarm service is created [2] — so there's a window where the DB says "running" but no service exists yet. If the PR is closed during that window, the deletion handler tries docker service rm on a service that doesn't exist yet, silently fails, deletes the DB record anyway [3], and then the worker finishes building and creates the Swarm service — now orphaned with no DB record.

The removeService() function returns errors instead of throwing them [4], and the caller doesn't check the result, so the failure is invisible.

There is no reconciliation mechanism. The daily Docker cleanup job only prunes stopped containers and unused images — it does not touch orphaned Swarm services [5].

Related prior issues:

• #2452 reports preview containers remaining after PR closure [6]
• Several PRs have addressed adjacent orphan container problems: PR fix: prevent orphaned containers when deleting compose services #4185 for Compose cleanup [7], PR fix: set FailureAction=rollback for swarm services default UpdateConfig #3810 for Swarm orphan containers [8], and PR fix: inject COMPOSE_PROJECT_NAME to prevent orphaned containers on redeploy #4188 for consistent COMPOSE_PROJECT_NAME [9] — but the core creation-vs-deletion race for previews remains unaddressed.

What would fix this:

1. Cancel or await in-flight creation jobs before deletion — store the BullMQ job ID in the preview record and check/cancel it during teardown.
2. Set status to "running" only after mechanizeDockerContainer() succeeds, not at worker start.
3. Add a periodic reconciliation job that compares preview_deployments records against docker service ls output and removes orphaned services.
4. Fix removeService() error handling — at minimum log the failure, and consider retrying service removal after a delay to catch late-created services.

Your host-side reconciliation script is a solid workaround in the meantime.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[budivoogt]

Follow-up after re-checking the history around #2452 / #2453:

• #2453 was helpful, but it looks like it only partially addressed the deletion path.
• Current v0.28.8 and current canary still set previewStatus = "running" before the preview service exists.
• Current canary still allows an in-flight preview build to continue after the preview deployment record has already been deleted.

That matches the controlled repro from our host:

1. open PR -> preview record created
2. close PR before the preview service exists
3. delete path removes the preview record
4. worker finishes later and creates the Swarm service anyway
5. service is now orphaned

I opened a draft PR with a minimal guard against that race here:

• fix: avoid preview service resurrection after delete #4209

The patch is intentionally small:

• skip preview queue jobs whose preview record is already gone
• re-check preview record existence immediately before mechanizeDockerContainer() in preview deploy/redeploy flows

It is not a full queue-cancellation refactor, but it should stop the specific “deleted preview resurrects into an orphan service” failure mode we reproduced.