`S3Downloader` cache races cause corrupt downloads under concurrent pipeline runs

[medsriha]

Describe the bug

When multiple pipeline runs execute in parallel, S3Downloader corrupts files in its on-disk cache, causing downstream components like DocumentToImageContent to fail with errors such as PDFium: Data format error, Failed to load page, or KeyError on a missing page.

The component shares one cache directory (file_root_path) across all concurrent run() calls, and two races compound - could be more:

1. _download_file writes straight to the final cache path, so a concurrent reader can observe a partial PDF mid-write
2. _cleanup_cache only protects files referenced by the current run, so it can delete files another in-flight run is about to read

To Reproduce

1. Wire S3Downloader → DocumentToImageContent in a pipeline.
2. Call pipeline.run() from multiple threads simultaneously, requesting documents whose total file count exceeds max_cache_size default 100. This forces cache eviction while runs are in flight.
3. Observe intermittent pdfium failures on DocumentToImageContent. Failure rate scales with parallelism.

[anakin87]

I've been investigating this bug a bit...

_download_file writes straight to the final cache path, so a concurrent reader can observe a partial PDF mid-write

I think that this can be fixed by writing the partial file to a temp location and then moving it

(I also found a separate minor race condition that I'll fix)

---

_cleanup_cache only protects files referenced by the current run, so it can delete files another in-flight run is about to read

In this case, I probably need more information:

• Are you running the same Pipeline multiple times concurrently? Or different pipelines writing to the same file_root_path?
• For parallelism, are you using multi-threading or multi-processing?

Depending on your scenario, we could implement different solutions. A simple one is exposing in the component a flag to disable cache. Another workaround, if you have different pipelines, might involve setting a different file_root_path.

I look forward to receiving information.

[medsriha]

Are you running the same Pipeline multiple times concurrently? Or different pipelines writing to the same file_root_path?

Same pipeline running multiple queries concurrently

For parallelism, are you using multi-threading or multi-processing?

This is running in the Jobs feature on the platform. @tstadel may know better.

[anakin87]

@sjrl suggested adding a flag to disable the cache (I think that setting max_cache_size to a big number would be roughly equivalent). Would this be an acceptable solution/workaround to prevent uninented deletion?

[anakin87]

• Some improvements have been implemented in fix: S3Downloader - improve concurrent behavior #3327
• Other concurrency problems can be avoided by setting max_cache_size to a high value

@medsriha confirmed that adopting this strategy seems to avoid race conditions, so I am closing this issue. Feel free to reopen it if the problems happen again.