Skip to content

FUSE client hot-upgrade: controlled drain, READY/NACK handover, teardown fixes#957

Merged
chuandew merged 8 commits into
dingodb:mainfrom
chuandew:verify/libfuse-hotupgrade-link
Jul 2, 2026
Merged

FUSE client hot-upgrade: controlled drain, READY/NACK handover, teardown fixes#957
chuandew merged 8 commits into
dingodb:mainfrom
chuandew:verify/libfuse-hotupgrade-link

Conversation

@chuandew

@chuandew chuandew commented Jun 30, 2026

Copy link
Copy Markdown
Member

Summary

This PR adds graceful hot-upgrade support for the DingoFS FUSE client.

The new process takes over the existing kernel FUSE connection by receiving the old process's /dev/fuse fd over UDS, replays the saved FUSE INIT request, and loads the dumped VFS state. The old process uses the libfuse controlled-drain APIs before checkpointing, so requests that have already been read by the old daemon are handled before the fd is handed over.

This is a graceful handover path, not a fully transactional hot-upgrade with rollback at every failure point. Failures before checkpoint can NACK/resume on the old process. After Stop(true) + Dump, the old VFS has already been torn down, so a later new-side mount/load failure still requires remount or manual recovery.

Main Changes

  • Free the saved FUSE INIT receive buffer after it is copied and replayed through libfuse.
  • Move the VFS handover state file out of /tmp and publish it atomically with temp file + fsync + rename + directory fsync.
  • Add the hot-upgrade handover path:
    • UDS fd passing for the live /dev/fuse fd.
    • saved INIT buffer transfer and replay in the new process.
    • old-side handover controller driven by SIGHUP plus UDS kPrepare validation.
    • controlled drain through libfuse pause/resume/wait-drained APIs.
    • checkpoint through VFSWrapper::Stop(true) and state dump/load.
    • READY/NACK protocol: checkpoint success sends kReadyToExit; drain/checkpoint-precondition failure sends kNack and resumes the old process.
  • Fix teardown lifecycle issues found during hot-upgrade testing:
    • use HandleGuard so Stop waits for active handle users and releases reader/writer resources outside the handle lock.
    • make VFSHub/MetaWrapper Stop idempotent.
    • add dedicated read-cleanup and write-background executors so teardown does not submit cleanup work to already-stopped executors.
  • Make kReadyToExit a one-way final notification. The new process no longer sends kAck, and the old process no longer waits for it after checkpoint.
  • Fix the drain-timeout statfs wakeup deadlock: the wakeup thread is stopped and detached on both success and timeout, so a statfs blocked behind a paused FUSE session cannot prevent kNack/ResumeReceive().

Commit Layout

  • free leaked FUSE INIT receive buffer
  • move VFS state file off /tmp and publish atomically
  • controlled drain + flush checkpoint + ACK/NACK handover handshake
  • Release handle resources in Stop() via HandleGuard
  • Hot-upgrade teardown: dedicated read-cleanup/write-bg executors + idempotent VFSHub/meta Stop
  • make READY one-way and remove ACK wait
  • detach statfs wakeup thread on drain timeout

Validation

  • cmake --build build --target test_fuse_upgrade -j2
  • build/bin/test/test_fuse_upgrade

Current result: 16 tests passed.

Notes

This PR depends on the libfuse controlled-drain APIs used by HandoverSessionImpl:

  • fuse_session_pause_receive
  • fuse_session_resume_receive
  • fuse_session_wait_drained

@chuandew chuandew force-pushed the verify/libfuse-hotupgrade-link branch from 6295f7a to a256bdd Compare July 2, 2026 06:39
@chuandew chuandew marked this pull request as ready for review July 2, 2026 06:39
@chuandew chuandew changed the title [WIP] FUSE client hot-upgrade: controlled drain + handover handshake + teardown lifecycle FUSE client hot-upgrade: controlled drain + handover handshake + teardown lifecycle Jul 2, 2026
@chuandew chuandew force-pushed the verify/libfuse-hotupgrade-link branch from a256bdd to 78134a1 Compare July 2, 2026 08:51
@chuandew chuandew changed the title FUSE client hot-upgrade: controlled drain + handover handshake + teardown lifecycle FUSE client hot-upgrade: controlled drain, READY/NACK handover, teardown fixes Jul 2, 2026
@chuandew chuandew force-pushed the verify/libfuse-hotupgrade-link branch from 78134a1 to f4f1224 Compare July 2, 2026 08:57
@chuandew chuandew force-pushed the verify/libfuse-hotupgrade-link branch from 5209ece to 3e02123 Compare July 2, 2026 11:28
@chuandew chuandew enabled auto-merge July 2, 2026 11:28
@chuandew chuandew added this pull request to the merge queue Jul 2, 2026
Merged via the queue into dingodb:main with commit fe71103 Jul 2, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant