Skip to content

feat: support concurrent chunk uploads#145

Merged
TorstenDittmann merged 2 commits into
mainfrom
concurrent-chunk-uploads-1-9-x-minimal
May 21, 2026
Merged

feat: support concurrent chunk uploads#145
TorstenDittmann merged 2 commits into
mainfrom
concurrent-chunk-uploads-1-9-x-minimal

Conversation

@TorstenDittmann
Copy link
Copy Markdown
Contributor

This PR updates the SDK to support concurrent chunk uploads.

@greptile-apps
Copy link
Copy Markdown

greptile-apps Bot commented May 21, 2026

Greptile Summary

This PR replaces the sequential chunked-upload loop with a ThreadPoolExecutor-based concurrent implementation (up to 8 workers). The first chunk is always uploaded serially so the server's assigned upload ID can be captured, then all remaining chunks are dispatched in parallel.

  • The previous file-handle leak (open() held for the lifetime of the upload) is fixed: input = None for path-sourced files, and each chunk opens its own handle via read_chunk.
  • Progress state (completed_count, uploaded_size, last_result, final_result) is protected by a Lock, but on_progress is called while that lock is held; a slow or I/O-bound callback will stall every other thread waiting to report progress.
  • The return-value determination (final_result or last_result) under concurrent completion — flagged in a prior review thread — remains unresolved.

Confidence Score: 4/5

The core concurrent upload logic is structurally sound, but the unresolved final-result ambiguity from a prior review still applies, and the on_progress callback is invoked while holding the progress lock.

The file-handle leak is fixed and the threading approach is reasonable. A previously flagged issue — where the returned result may not reflect the server's completion state when the finishing chunk arrives concurrently — remains unaddressed. The on_progress-under-lock pattern can silently degrade parallelism if callers do any work in the callback.

appwrite/client.py — specifically the final-result selection logic and the progress-lock scope

Important Files Changed

Filename Overview
appwrite/client.py Refactors chunked upload to use ThreadPoolExecutor (8 workers). First chunk uploaded sequentially to get upload ID; remaining chunks uploaded concurrently. Fixes the previous file-handle leak. The on_progress callback is invoked while holding progress_lock, which can serialise in-flight uploads; and the final-result determination under concurrent completion (flagged in a previous thread) is still unresolved.

Reviews (2): Last reviewed commit: "feat: support concurrent chunk uploads" | Re-trigger Greptile

Comment thread appwrite/client.py
Comment thread appwrite/client.py Outdated
@TorstenDittmann TorstenDittmann merged commit c47e7c1 into main May 21, 2026
1 check passed
@TorstenDittmann TorstenDittmann deleted the concurrent-chunk-uploads-1-9-x-minimal branch May 21, 2026 17:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants