Copying between different filesystems with unrepresentable directory structure

[ap--]

Hi @martindurant,

In universal-pathlib a user reported an incomplete copy operation when copying between s3 and local.
Turns out they are basically trying to copy the following structure to a local filesystem, which can't be represented.

s3://bucket/key  # data stored in object under 'key'
s3://bucket/key/other-key-0
s3://bucket/key/other-key-1

I wonder if this issue came up when developing AbstractFileSystem.get or fsspec.generic.rsync and what an fsspec native solution for this could look like.

For UPath, I am currently working on exposing a callback, that allows the user to decide how to handle the fs.isfile('bucket/key') is True and fs.isdir('bucket/key') is True collision. I'd be happy to add an equivalent mechanism to fsspec.generic if that's a good solution.

Ideally, there would be some way that allows the filesystem class to tell if it supports a key / objectstore-like storage mechanism where directories are virtual or if a given path can't be a file and a dir at the same time.

Cheers,
Andreas

[martindurant]

This has never come up :|

I suppose this should indeed be handled in generic or any fs.get() for the key-value implementations.

We should probably provide one or more default behaviours, though, such as adding a suffix to the file (as opposed to directory) name.

Ideally, there would be some way that allows the filesystem class to tell if it supports a key / objectstore-like storage mechanism where directories are virtual or if a given path can't be a file and a dir at the same time.

Yeah, this doesn't exist. I wonder if there are any ambiguous cases.