Equivalent to pyfilesystem2's MountFS?

[dAnjou]

Hi,

just wanted to make sure I'm not completely missing something before I have a go at this myself. Is there an equivalent to https://pyfilesystem2.readthedocs.io/en/latest/reference/mountfs.html?

Thanks 🙏

[martindurant]

We have dirFS for prefixes, but not a complete mountFS.

You might also want to look within fsspec.generic for code around efficient inter-FS operations.

[dAnjou]

Thank you for the quick response!

I did have a look at DirFileSystem, and if I understand correctly, it gives me a virtual filesystem that is a level deeper from the given filesystem.

What I need is a virtual filesystem that is a level higher, where I can have folders that contain other filesystems.

For now, it wasn't too hard to port the functionalities of pyfilesystem2's MountFS that I need in my codebase, which kinda look like this:

import re
from pathlib import PurePosixPath
from fsspec.implementations.memory import MemoryFileSystem

class MountFileSystem(AbstractFileSystem):
    def __init__(self, *args, asynchronous=False, loop=None, batch_size=None, **kwargs):
        super().__init__(*args, asynchronous=False, loop=None, batch_size=None, **kwargs)
        self.default_fs = MemoryFileSystem()
        self.mounts = {}

    def _delegate(self, path):
        _path = PurePosixPath(path)
        for mount_path, fs in self.mounts.items():
            if _path.is_relative_to(mount_path):
                return fs, str(PurePosixPath("/") / _path.relative_to(mount_path))

        return self.default_fs, path

    def mount(self, path, fs):
        self.default_fs.makedirs(abspath(normpath(path)), exist_ok=True)
        self.mounts[PurePosixPath(abspath(normpath(path)))] = fs

    def ls(self, path, detail=True, **kwargs):
        fs, _path = self._delegate(abspath(normpath(path)))
        return fs.ls(_path, detail=detail, **kwargs)

    # TODO: implement other methods here, similar to DirFileSystem

def normpath(path):
    if path in ("", "/"):
        return path

    # An early out if there is no need to normalize this path
    if not re.compile(r"(^|/)\.\.?($|/)|//").search(path):
        return path.rstrip("/")

    prefix = "/" if path.startswith("/") else ""
    components = []
    for component in path.split("/"):
        if component not in (".", ""):
            components.append(component)
    return prefix + "/".join(components)

def abspath(path):
    if not path.startswith("/"):
        return "/" + path
    return path

It's neither pretty nor complete, but it does the job for my use case 😅

[martindurant]

What I need is a virtual filesystem that is a level higher, where I can have folders that contain other filesystems.

Yes, I understood - and if you would like the complete your implementation and submit it, it would probably be useful for others. You would need to decide whether to make it async (in case any of the backends is async) or only sync (and miss out on any async benefits when they are available).