Implement per-object locks for better parallelism in filesystem backend

[misha-drozd]

Fixes #1978 for filesystem backend

[fsouza]

DeleteObject and ComposeObject are very racy in this version of the PR. Can you take a look?

[fsouza]

ComposeObject still has a race condition. It reads source objects one by one (acquiring and releasing each read lock), then creates the destination. Between reading sources and creating the destination, another goroutine could delete/modify a source object.

func (s *storageFS) ComposeObject(...) (StreamingObject, error) {
    for _, n := range objectNames {
        obj, err := s.GetObject(bucketName, n)  // acquires & releases read lock
        // ...
    }
    // No locks held here! Source objects could be modified/deleted
    
    result, err := s.CreateObject(dest, NoConditions{})  // acquires dest write lock
}

To fix this properly, ComposeObject would need to acquire read locks on all source objects AND a write lock on the destination, in a consistent order (e.g., sorted by name) to avoid deadlocks, and hold them all until the operation completes.

Alternatively, ComposeObject could keep using the global lock for simplicity since it's a less common operation.

[misha-drozd]

To fix this properly, ComposeObject would need to acquire read locks on all source objects AND a write lock on the destination, in a consistent order (e.g., sorted by name) to avoid deadlocks, and hold them all until the operation completes.

Fixed.

I had a look around more and fixed a small bug in DeleteBucket: os.RemoveAll might fail if another object is being created while it is running (as it does "listdir" + "foreach(file) rm" inside - here's a tiny race window).

Another potential issue I left "as is" for now: GetObjectWithGeneration might be racy if between GetObject and reading the body the object is updated: as file handle is opened lazily, there's TOCTOU mismatch.

[misha-drozd]

Fixed lint issue.

[fsouza]

Thank you for addressing my previous comments. I gave this another pass (with the help of AI, I'll admit :D), and have two remaining concerns (well, 3 if you include a minor comment-only suggestion here):

ListObjects can observe partially-written/deleted objects

ListObjects holds the global RLock, but CreateObject and DeleteObject now use only per-object locks for the actual file I/O. This means ListObjects can run concurrently with object mutations after the brief global lock is released.

In createObject, the content file is written first (writeFile), then metadata (s.mh.write). If filepath.Walk inside listObjects sees the content file before metadata is written, getObjectAttrs will fail to read metadata, and the entire ListObjects call returns an error.

Similarly, DeleteObject removes metadata first (s.mh.remove), then the file (os.Remove). A concurrent ListObjects could see the file but fail on missing metadata.

The original code prevented this because CreateObject/DeleteObject held the global write lock for the entire duration, blocking ListObjects's RLock.

Any thoughts on how we can fix this? One idea is to have list only look at metadata files, or ignore errors when it finds a file but cannot find the metadata. Let me know what you think.

Test helper duplication

bytesReadSeekCloser / newStreamingContent in the test file duplicate the existing noopSeekCloser type in internal/backend/object.go. You can replace usages with:

noopSeekCloser{bytes.NewReader(data)}

Unbounded objectLocks growth

The objectLocks map grows without bound since locks are never removed (except in DeleteAllFiles). I'm fine with this tradeoff — it keeps the code simpler and the per-entry cost is small for typical test workloads. Do you think it's worth adding a short comment on the objectLocks field noting this intentional behavior? Something like:

// objectLocks holds per-object locks. Entries are never removed to avoid
// races on lock deletion; memory growth is negligible for typical usage.
objectLocks map[string]*sync.RWMutex

[misha-drozd]

Any thoughts on how we can fix this?

Maybe eagerly walk and open the file handles while ListObjects is holding the lock? I have already been bitten once by lazy file handle opening earlier in this change.