Description
When running gcsfs in highly concurrent environments—specifically utilizing multiprocessing processes alongside internal multithreading—the underlying fsspec cache logic cannot handle when multiple threads try to instantiate concurrently.
If a Python process forks (e.g., via multiprocessing.Pool), the fsspec child process clears its inherited instance cache (os.getpid() != cls._pid). However, because this cache-clearing check and instantiation process was not thread-safe:
- All threads evaluate the newly empty cache post-fork.
- They bypass the cache concurrently.
- Every single overlapping thread instantiates a new, fully independent
GCSFileSystem instance and overwrites the local cache.
This leads to significant redundant filesystem instantiations and associated overhead due to multiple parallel clients standing up unnecessarily.
Proposed Solution
We have developed a complete patch and will be landing a Pull Request in fsspec that introduces:
- Double-checked
threading.Lock locking directly inside _Cached.__call__.
- Safe boundaries for
os.getpid() != cls._pid post-fork cache clearing.
- Fixing
tokenize() dependencies on PIDs prior to evaluating the cache.
Description
When running
gcsfsin highly concurrent environments—specifically utilizingmultiprocessingprocesses alongside internal multithreading—the underlyingfsspeccache logic cannot handle when multiple threads try to instantiate concurrently.If a Python process forks (e.g., via
multiprocessing.Pool), thefsspecchild process clears its inherited instance cache (os.getpid() != cls._pid). However, because this cache-clearing check and instantiation process was not thread-safe:GCSFileSysteminstance and overwrites the local cache.This leads to significant redundant filesystem instantiations and associated overhead due to multiple parallel clients standing up unnecessarily.
Proposed Solution
We have developed a complete patch and will be landing a Pull Request in
fsspecthat introduces:threading.Locklocking directly inside_Cached.__call__.os.getpid() != cls._pidpost-fork cache clearing.tokenize()dependencies on PIDs prior to evaluating the cache.