Skip to content

fix(watcher): serialize watch and close on the native watcher#14757

Open
stormslowly wants to merge 3 commits into
mainfrom
fix/native-watcher-close-race-macos
Open

fix(watcher): serialize watch and close on the native watcher#14757
stormslowly wants to merge 3 commits into
mainfrom
fix/native-watcher-close-race-macos

Conversation

@stormslowly

@stormslowly stormslowly commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Why

On macOS, tearing down and restarting the native watcher (experiments.nativeWatcher) intermittently aborts the process:

*** Terminating app due to uncaught exception 'NSInvalidArgumentException',
    reason: '-[OS_os_log addObject:]: unrecognized selector sent to instance ...'
libc++abi: terminating due to uncaught exception of type NSException  →  SIGABRT

Symbolicated crash stack (release-debug build), on a tokio worker thread:

notify::fsevent::FsEventWatcher::append_path        // CFArrayAppendValue(self.paths, ...)
notify::fsevent::FsEventWatcher::watch
rspack_watcher::disk_watcher::DiskWatcher::watch
rspack_watcher::FsWatcher::wait_for_event
rspack_binding_api::native_watcher::NativeWatcher::watch   // spawned task

A use-after-free between two entry points that were never kept apart:

  • watch() spawns a detached task and returns to JS immediately. The task registers paths with FSEvents via append_path, which mutates the watcher's CFMutableArray (self.paths).
  • close() drops the watcher — DropCFRelease(self.paths).
  • They land on the same instance: NativeWatchFileSystem memoizes one NativeWatcher per compiler and clears #inner only after close()'s promise resolves. The # Safety note on close asserted callers must not overlap the two, but nothing enforced it.

So append_path writes into the CFArray while close frees it. The freed slot gets reused by an os_log object, and the next CFArrayAppendValue dispatches addObject: to a non-array → NSException → SIGABRT.

Where the window is. DiskWatcher::watch calls into notify only for patterns it isn't already watching, and on macOS WatcherRootAnalyzer collapses every dependency into a single recursive common root that is stable across rebuilds. append_path therefore runs on a watcher's first watch() — one recursive FSEvents subscription over the whole project root, which is far from instantaneous — while steady-state rebuilds dedupe to a no-op. The race is a close() (or a second watch(), which would also see an empty pattern set) landing inside that first registration.

Instrumenting a real consumer confirms it is reachable: Rstest rebuilds its dev server whenever a config file changes, and its watch-mode restart test edits the config while startup is still in progress. close() arrives 3 ms after the first watch() on the same memoized NativeWatcher, and the process aborts in append_path.

This is distinct from the analyzer/root.rs path-tree assert — it still reproduces with that fix applied.

What

Serialize the two entry points that reach the disk watcher with an async mutex:

  • a watch task acquires it before touching the watcher, and holds it across the whole watcher.watch();
  • close aborts the in-flight watch task, then acquires the same mutex before tearing the watcher down.

watch and close are the only paths that reach DiskWatcher (FsWatcher::watchwait_for_eventdisk_watcher.watch(), and FsWatcher::closedisk_watcher.close()). pause and trigger_event take &self and only touch the executor / trigger, never the FSEvents state, so they stay outside the lock — the # Safety note is updated to say exactly that.

Cancellation-safe by construction: a future can only be dropped at an .await, never mid-FFI, so an aborted task always releases the guard with the watcher in a consistent state. Holding the mutex in close is proof that no task is inside append_path. It cannot deadlock either — abort() always precedes the lock wait, so the current holder is guaranteed to reach an await and release.

Also adds rspack_napi::runtime::spawn_handle, which returns the JoinHandle needed to abort (the existing spawn drops it).

Rejected alternative (raised in review): have each new watch task abort() and await its predecessor. Not cancellation-safe — close aborting the head of the chain cancels it at its prev.await, which drops the predecessor's JoinHandle and detaches it; the abandoned tasks keep running into append_path while close drops the watcher. The test below reproduced both the resulting SIGSEGV and a hang (notify's stop() spinning forever on CFRunLoopIsWaiting against corrupted state), which is why this uses a lock.

Test

tests/rspack-test/NativeWatcherLifecycle.test.js drives binding.NativeWatcher directly: overlapping watch tasks on a fresh watcher followed by an immediate close, repeated. The file set is large enough that the work preceding append_path (path-manager update, mtime baselines, path-tree rebuild) keeps the registration in flight long enough for the close to land on it. It runs in the nativeWatcher project (retry: 0, maxConcurrency: 1) so a native abort cannot be masked by a retry.

Verification

macOS arm64, binding built from this branch:

pre-fix abort+await chain this PR (mutex)
NativeWatcherLifecycle 6/6 crash 4/6 fail (3 hang, 1 SIGSEGV) 6/6 pass
NativeWatcher.part1 122/122
NativeWatcher.part2 55/55
NativeWatcher.part3 60/60

part1/2/3 matter because the watch task now holds the mutex across the whole watch(), including the long event-await — they confirm real watch cycles are not stalled by it.

@github-actions

github-actions Bot commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

📦 Binary Size-limit

Comparing eb56f24 to fix(fs): include paths in native filesystem errors (#14760) by hardfist

❌ Size increased by 4.00KB from 66.51MB to 66.51MB (⬆️0.01%)

@codspeed-hq

codspeed-hq Bot commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 43 untouched benchmarks
⏩ 47 skipped benchmarks1


Comparing fix/native-watcher-close-race-macos (eb56f24) with main (cfa680e)

Open in CodSpeed

Footnotes

  1. 47 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@stormslowly stormslowly marked this pull request as ready for review July 13, 2026 02:39
@stormslowly stormslowly requested review from Copilot and hardfist July 13, 2026 02:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses a macOS native watcher crash caused by a race between an in-flight watch task mutating FSEvents state and a concurrent close that frees that state. It does so by tracking the spawned watch task and coordinating task cancellation/teardown before dropping the underlying watcher.

Changes:

  • Add rspack_napi::runtime::spawn_handle to return a JoinHandle for spawned tasks.
  • Track the native watcher’s in-flight watch task and abort it before starting a new watch cycle.
  • Make close() abort and await the in-flight watch task before closing the underlying FsWatcher.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
crates/rspack_napi/src/runtime.rs Adds spawn_handle() so callers can retain and manage spawned task JoinHandles.
crates/rspack_binding_api/src/native_watcher.rs Stores the watch task handle and coordinates cancellation during re-watch and close to avoid macOS FSEvents teardown races.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread crates/rspack_binding_api/src/native_watcher.rs
@stormslowly stormslowly marked this pull request as draft July 13, 2026 03:34
`NativeWatcher::watch` spawns a detached task that drives the FSEvents backend's
`append_path` (mutating the watcher's CFMutableArray), while `NativeWatcher::close`
drops the watcher (`CFRelease` of that array). Nothing kept the two apart, so the
close-old-then-watch-new restart flow could free the CFArray while a watch task was
still appending to it — a use-after-free that aborts the process on macOS.

Guard every access to the watcher with an async mutex: a watch task acquires it
before touching the watcher and holds it for the whole `watch()`, and `close`
aborts the in-flight task and then acquires the same mutex before tearing the
watcher down. A future can only be cancelled at an `.await`, never mid-FFI, so an
aborted task always releases the guard with the watcher in a consistent state, and
holding the mutex in `close` proves no task is inside `append_path`.

Adds `rspack_napi::runtime::spawn_handle` to retain the JoinHandle needed to abort.
Drives `binding.NativeWatcher` directly: several overlapping watch tasks on one
watcher followed by an immediate close, repeated. Reproduces the use-after-free
(SIGSEGV) on every run without the fix.
@stormslowly stormslowly force-pushed the fix/native-watcher-close-race-macos branch from c4db59c to b082901 Compare July 13, 2026 07:48
@stormslowly stormslowly changed the title fix(watcher): abort in-flight watch task before closing the native watcher fix(watcher): serialize watch and close on the native watcher Jul 13, 2026
@stormslowly stormslowly marked this pull request as ready for review July 13, 2026 08:22
@stormslowly stormslowly enabled auto-merge (squash) July 14, 2026 05:05

@hardfist hardfist Jul 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why need spawn_handle here and what does spawn_handle mean here?

* This function is unsafe because it uses `&mut self` to call the watcher asynchronously.
* It's important to ensure that the watcher is not used in any other places before this function is finished.
* You must ensure that the watcher not call watch, close or pause in the same time, otherwise it may lead to undefined behavior.
* `watch` and `close` are serialized internally, but calling `pause` while this function is

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so calling pause while watching still can cause panic | abort issue?

* This function is unsafe because it uses `&mut self` to call the watcher asynchronously.
* It's important to ensure that the watcher is not used in any other places before this function is finished.
* You must ensure that the watcher not call watch, close or pause in the same time, otherwise it may lead to undefined behavior.
* `watch` and `close` are serialized internally, but calling `pause` while this function is

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this api's semantic is not same as js version

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants