fix(macOS): keep the GUI responsive when many files are added to a sync folder#10370
Open
ftapps wants to merge 4 commits into
Open
fix(macOS): keep the GUI responsive when many files are added to a sync folder#10370ftapps wants to merge 4 commits into
ftapps wants to merge 4 commits into
Conversation
The FSEvents stream was scheduled on the main dispatch queue, so every event batch was handled on the GUI thread. Dropping a large tree into the sync folder made that work unbounded: addCoalescedPaths() walked each reported directory's whole subtree with QDirIterator and deduplicated it with a linear QStringList::contains(), and on kFSEventStreamEventFlagMustScanSubDirs or a kernel drop -- exactly what a mass copy triggers -- notifyAll() enumerated the entire sync folder and reported every path one by one. Each reported path then costs a journal lookup and several stats in Folder::slotWatchedPathChanged(), so the window stayed frozen until the whole batch was processed. Deliver events on a private serial queue instead, as the Windows backend already does with its own WatcherThread, and hand each batch over to the FolderWatcher's thread with a queued invocation. Deduplicate the expansion with a QSet rather than a linear scan, and stop expanding past maxCoalescedPaths: nothing is lost by stopping, because the directory being expanded was itself reported and SyncEngine::shouldDiscoverLocally() descends into a touched directory's entire subtree. notifyAll() no longer walks the tree either: a full local discovery re-reads it anyway, so it just emits lostChanges(). It reports no path deliberately -- the watched root would reach the local discovery tracker as an empty relative path, which matches only the root itself and not the tree below it. Signed-off-by: Felipe Tumonis <ftumonis@gmail.com>
slotNextSyncFullLocalDiscovery() is documented to ensure the next sync performs a full local discovery, but it only invalidated _timeSinceLastFullLocalDiscovery. A sync that was already running restarts that timer when it finishes, so a request arriving mid-sync -- the common case, since the watcher emits lostChanges() precisely while changes are pouring in -- was silently dropped and the next sync read from the database instead. The missed changes then waited for the periodic full local discovery, an hour by default, or forever where that interval is configured negative. Track whether a request arrived after the running sync fixed its discovery style, and only mark a full local discovery as done when none did. While here, check the selective sync list in warnOnNewExcludedItem() before stat()ing the file: the function runs once per added file and the list is empty unless the user configured selective sync. Signed-off-by: Felipe Tumonis <ftumonis@gmail.com>
slotItemDiscovered() kept _syncItems sorted by inserting each item at its lower_bound. Inserting into the middle of a contiguous vector memmoves the tail, so discovering n items costs O(n^2): for a first sync of a large tree that is billions of shared-pointer moves on the GUI thread, all of it before the first file is transferred. Nothing reads _syncItems before discovery ends -- the first consumer is handleMassDeletion(), from slotDiscoveryFinished() -- so append and sort once there. Stable, so that items comparing equal keep the order they were discovered in. Signed-off-by: Felipe Tumonis <ftumonis@gmail.com>
getSelectiveSyncList() runs a full query and materializes the whole list on every call, and Folder::warnOnNewExcludedItem() calls it once per file added to the sync folder. The list only changes when the user edits the selective sync configuration, so keep it in memory and drop it in setSelectiveSyncList(), the only writer of the table, and on close(). Signed-off-by: Felipe Tumonis <ftumonis@gmail.com>
1d576a9 to
0de37dc
Compare
claucambra
reviewed
Jul 16, 2026
claucambra
left a comment
Collaborator
There was a problem hiding this comment.
Hi @ftapps how did you reproduce the issue before the fix and verify it worked after the fact? Would like to test on my own system
If possible it would also be good to formalise this into a proper test so we can guard against regressions on this :)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On macOS the client's window freezes for as long as it takes to process a large
batch of files, in two situations users hit routinely:
The sync itself is fine — it is the GUI that stops responding until the initial
processing is over.
Cause
FolderWatcherPrivate::startWatching()scheduled the FSEvents stream ondispatch_get_main_queue(), so the whole notification pipeline ran on the GUIthread, unlike the Windows backend which uses its own
WatcherThread. The workdone there is unbounded in the number of files:
addCoalescedPaths()walked every reported directory's whole subtree withQDirIteratorand deduplicated it with a linearQStringList::contains();notifyAll()— reached onkFSEventStreamEventFlagMustScanSubDirsor akernel drop, which is exactly what a mass copy triggers — enumerated the
entire sync folder and reported every path individually;
Folder::slotWatchedPathChanged().The setup case has a separate, cross-platform cause:
SyncEngine::slotItemDiscovered()kept
_syncItemssorted with alower_bound+QVector::insertper item, whichmemmoves the tail every time and costs O(n²) overall, all of it on the GUI thread
before the first byte is transferred.
Changes
Four self-contained commits:
fix(macOS): handle file system events off the GUI thread— private serialdispatch queue, batches handed to the watcher's thread with a queued
invocation; subtree expansion bounded and deduplicated with a
QSet;notifyAll()asks for a full local discovery instead of walking the tree.fix: keep a full local discovery request across a running sync— a requestarriving mid-sync was silently dropped, because the running sync restarts
_timeSinceLastFullLocalDiscoverywhen it finishes. This one is pre-existingand not macOS-specific, but the change above depends on the request being
honoured.
perf: sort discovered sync items once instead of inserting in order— O(n²) → O(n log n).perf: cache the selective sync list in the journal— it was queried once per added file.Testing
Built on macOS (Qt 6) and run as the daily driver against a real server, on a
sync folder of ~84 GB.
FolderWatcherTest,LocalDiscoveryTest,SyncJournalDBTest,SyncMoveTest,RemoteDiscoveryTest,SyncConflictTest,SyncFileStatusTrackerTest,AllFilesDeletedTest,BlacklistTest,SyncDeleteTestall pass. A debugbuild covers commit 3 through the
Q_ASSERT(std::is_sorted(...))infinishSync()andOwncloudPropagator::start().testMove3LevelDirWithFilein particular still passes: FSEvents does notreport the items below a renamed directory, so the expansion is still needed --
it is bounded, not removed.
stayed responsive throughout, where it used to freeze.
sample(1)on the mainthread during discovery shows it parked in
nextEventMatchingMaskwith noFolderWatcher/ProcessDirectoryJobframes on it.released 33.0.7 build alongside the patched one, against the same account and
folder. Activity Monitor showed the released build as "Not Responding" while
the patched one kept its main thread ~80% idle under the same load.
Notes for reviewers
addCoalescedPaths()deliberately bounds only its own expansion, not theincoming batch: those are events FSEvents actually reported, they are capped by
the kernel's own buffer, and reporting them drives things a full local
discovery cannot redo, such as releasing office file locks.
changesLost()reports no path on purpose. The watched root reaches the localdiscovery tracker as an empty relative path, and
{""}matches only the rootitself, not the tree below it — so it would look like a fix while quietly
discovering nothing.
removes the unbounded single blocks, so the remaining work yields to the event
loop between batches.