Skip to content

Commit 6e3d406

Browse files
adityaoberaiadityaoberai
authored
Merge pull request #2966 from appwrite/feat-fatser-uploads
new blog for parallel chunk uploads
2 parents d7b4ccc + 30b01ba commit 6e3d406

8 files changed

Lines changed: 388 additions & 5 deletions

File tree

‎.optimize-cache.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -601,6 +601,7 @@
601601
"static/images/blog/everything-new-with-appwrite-1.5/1.5-recap.png": "1d3c646f6902757152d98861630c1952631a54f222af7f8476f53f4d0d3c59f2",
602602
"static/images/blog/everything-new-with-appwrite-1.5/messaging-console.png": "769b7df74c9107a5ccacfe87722293adbfbd91ab702c79b03838c2368e9971ac",
603603
"static/images/blog/examples-of-vibe-coding/cover.png": "745d0e65c7981fe852b2e1797c3163cd4e4c147227b906cf305019137cb4624f",
604+
"static/images/blog/faster-storage-uploads-parallel-chunks/cover.png": "4565a9b19b4cafad3ed9ca3f2d4c6e4437379c5d10de6bad64dc1fb85590e49b",
604605
"static/images/blog/february-and-march-product-update-realtime-queries-appwrite-skills-and-new-database-features/Announcing_Appwrite_Skills__Give_your_AI_agents_Appwrite_expertise.png": "f6556f4786b55f53d06ca4c1a74ce0e488fa898099bf6458cab3e525f0a05d54",
605606
"static/images/blog/february-and-march-product-update-realtime-queries-appwrite-skills-and-new-database-features/Announcing_Realtime_Channel_helpers__Type-safe_subscriptions_made_simple.png": "a937f5b617fcbaa1d8d6af38f061f132a7590b85f93b104ee22119e80d5ed6d2",
606607
"static/images/blog/february-and-march-product-update-realtime-queries-appwrite-skills-and-new-database-features/comm_recoggg.png": "207e8acd544ebdd118f9aafb9d049dfa7fbb87868947d3df8b0fe86288848df0",

‎src/lib/generated/github-stars.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
{
2-
"stars": 55973,
3-
"fetchedAt": "2026-05-04T21:03:24.879Z"
2+
"stars": 55982,
3+
"fetchedAt": "2026-05-05T10:41:28.857Z"
44
}

‎src/routes/blog/post/faster-storage-uploads-parallel-chunks/+page.markdoc‎

Lines changed: 363 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,363 @@
1+
---
2+
layout: post
3+
title: "Up to 7x faster Appwrite Storage uploads with parallel chunks"
4+
description: Appwrite SDKs now upload Storage chunks in parallel on runtimes with native concurrency. Our Node SDK benchmarks show up to 7.10x faster uploads on large files, with no API changes.
5+
date: 2026-05-21
6+
cover: /images/blog/faster-storage-uploads-parallel-chunks/cover.avif
7+
timeToRead: 9
8+
author: eldad-fux
9+
category: announcement
10+
featured: false
11+
---
12+
13+
Uploading large files to [Appwrite Storage](/docs/products/storage) should feel snappy on a good connection and **bearable on real-world networks**. Until now, SDKs typically sent file chunks one after another. That approach is simple and reliable, but it leaves performance on the table. Each chunk waits for the previous one, so you rarely saturate bandwidth or use the browser's ability to run several HTTP requests in parallel.
14+
15+
We are releasing updated **Appwrite SDKs** that upload **multiple chunks at the same time**, with defaults tuned for how browsers and HTTP clients actually behave. The speedup scales with file size: in our Node SDK benchmarks, a 1.28 GB upload dropped from **4 minutes 44 seconds to under 40 seconds, a 7.10x improvement** at the default concurrency of 8.
16+
17+
# Why sequential chunks cap your speed
18+
19+
Chunked uploads exist so large files do not need to live entirely in memory and so failures can be retried at chunk granularity. The tradeoff is that strictly sequential chunking means:
20+
21+
- **Underused bandwidth** - the connection often sits idle while the client waits on the next chunk.
22+
- **Latency stacked in series** - every chunk pays its own round trip one after another.
23+
- **Browser limits left unused** - browsers allow multiple connections per origin, but a single in-flight upload does not use that capacity.
24+
25+
The server must accept chunks in a well-defined way and assemble a complete file safely. Making uploads parallel required both smarter clients and a backend that could handle out-of-order and concurrent chunk writes without corrupting metadata or final assembly.
26+
27+
# What we shipped
28+
29+
The same establish-then-parallelize pattern is implemented everywhere the host runtime supports it. Each SDK uses that language's native concurrency primitives so multiple chunk requests run together without blocking each other on the wire. The updated SDKs:
30+
31+
- Establish the upload by sending the first chunk in a controlled way to obtain an upload identifier.
32+
- Upload the remaining chunks concurrently up to a fixed maximum parallelism.
33+
34+
The examples below are the same `createFile` calls you use today. Point them at a large file, bump the SDK and server, and uploads get faster with no extra parameters for parallelism.
35+
36+
{% multicode %}
37+
```client-web
38+
import { Client, Storage, ID } from 'appwrite';
39+
40+
const client = new Client()
41+
.setEndpoint('https://<REGION>.cloud.appwrite.io/v1')
42+
.setProject('<PROJECT_ID>');
43+
44+
const storage = new Storage(client);
45+
46+
const uploaded = await storage.createFile({
47+
bucketId: 'videos',
48+
fileId: ID.unique(),
49+
file: document.getElementById('uploader').files[0]
50+
});
51+
```
52+
```client-flutter
53+
import 'package:appwrite/appwrite.dart';
54+
55+
final client = Client()
56+
.setEndpoint('https://<REGION>.cloud.appwrite.io/v1')
57+
.setProject('<PROJECT_ID>');
58+
59+
final storage = Storage(client);
60+
61+
final uploaded = await storage.createFile(
62+
bucketId: 'videos',
63+
fileId: ID.unique(),
64+
file: InputFile.fromPath(
65+
path: './large-video.mp4',
66+
filename: 'large-video.mp4',
67+
),
68+
);
69+
```
70+
```client-react-native
71+
import { Client, Storage, ID } from 'react-native-appwrite';
72+
73+
const client = new Client()
74+
.setEndpoint('https://<REGION>.cloud.appwrite.io/v1')
75+
.setProject('<PROJECT_ID>');
76+
77+
const storage = new Storage(client);
78+
79+
const uploaded = await storage.createFile({
80+
bucketId: 'videos',
81+
fileId: ID.unique(),
82+
file: {
83+
name: 'large-video.mp4',
84+
type: 'video/mp4',
85+
size: fileSize,
86+
uri: fileUri
87+
}
88+
});
89+
```
90+
```client-apple
91+
import Appwrite
92+
93+
let client = Client()
94+
.setEndpoint("https://<REGION>.cloud.appwrite.io/v1")
95+
.setProject("<PROJECT_ID>")
96+
97+
let storage = Storage(client)
98+
99+
let uploaded = try await storage.createFile(
100+
bucketId: "videos",
101+
fileId: ID.unique(),
102+
file: InputFile.fromPath("./large-video.mp4")
103+
)
104+
```
105+
```client-android-kotlin
106+
import io.appwrite.Client
107+
import io.appwrite.ID
108+
import io.appwrite.models.InputFile
109+
import io.appwrite.services.Storage
110+
111+
suspend fun upload() {
112+
val client = Client(applicationContext)
113+
.setEndpoint("https://<REGION>.cloud.appwrite.io/v1")
114+
.setProject("<PROJECT_ID>")
115+
116+
val storage = Storage(client)
117+
118+
val uploaded = storage.createFile(
119+
bucketId = "videos",
120+
fileId = ID.unique(),
121+
file = InputFile.fromPath("./large-video.mp4")
122+
)
123+
}
124+
```
125+
```server-nodejs
126+
import { Client, Storage, ID } from 'node-appwrite';
127+
import { InputFile } from 'node-appwrite/file';
128+
129+
const client = new Client()
130+
.setEndpoint('https://<REGION>.cloud.appwrite.io/v1')
131+
.setProject('<PROJECT_ID>')
132+
.setKey('<API_KEY>');
133+
134+
const storage = new Storage(client);
135+
136+
const uploaded = await storage.createFile({
137+
bucketId: 'videos',
138+
fileId: ID.unique(),
139+
file: InputFile.fromPath('./large-video.mp4', 'large-video.mp4')
140+
});
141+
```
142+
```server-python
143+
from appwrite.client import Client
144+
from appwrite.services.storage import Storage
145+
from appwrite.id import ID
146+
from appwrite.input_file import InputFile
147+
148+
client = Client()
149+
client.set_endpoint('https://<REGION>.cloud.appwrite.io/v1')
150+
client.set_project('<PROJECT_ID>')
151+
client.set_key('<API_KEY>')
152+
153+
storage = Storage(client)
154+
155+
uploaded = storage.create_file(
156+
bucket_id='videos',
157+
file_id=ID.unique(),
158+
file=InputFile.from_path('./large-video.mp4'),
159+
)
160+
```
161+
```server-dart
162+
import 'package:dart_appwrite/dart_appwrite.dart';
163+
164+
final client = Client()
165+
.setEndpoint('https://<REGION>.cloud.appwrite.io/v1')
166+
.setProject('<PROJECT_ID>')
167+
.setKey('<API_KEY>');
168+
169+
final storage = Storage(client);
170+
171+
final uploaded = await storage.createFile(
172+
bucketId: 'videos',
173+
fileId: ID.unique(),
174+
file: InputFile.fromPath(
175+
path: './large-video.mp4',
176+
filename: 'large-video.mp4',
177+
),
178+
);
179+
```
180+
```server-php
181+
<?php
182+
183+
use Appwrite\Client;
184+
use Appwrite\ID;
185+
use Appwrite\InputFile;
186+
use Appwrite\Services\Storage;
187+
188+
$client = (new Client())
189+
->setEndpoint('https://<REGION>.cloud.appwrite.io/v1')
190+
->setProject('<PROJECT_ID>')
191+
->setKey('<API_KEY>');
192+
193+
$storage = new Storage($client);
194+
195+
$uploaded = $storage->createFile(
196+
bucketId: 'videos',
197+
fileId: ID::unique(),
198+
file: InputFile::withPath('./large-video.mp4'),
199+
);
200+
```
201+
```server-ruby
202+
require 'appwrite'
203+
204+
include Appwrite
205+
206+
client = Client.new
207+
.set_endpoint('https://<REGION>.cloud.appwrite.io/v1')
208+
.set_project('<PROJECT_ID>')
209+
.set_key('<API_KEY>')
210+
211+
storage = Storage.new(client)
212+
213+
uploaded = storage.create_file(
214+
bucket_id: 'videos',
215+
file_id: ID.unique,
216+
file: InputFile.from_path('./large-video.mp4'),
217+
)
218+
```
219+
```server-go
220+
package main
221+
222+
import (
223+
"github.com/appwrite/sdk-for-go/client"
224+
"github.com/appwrite/sdk-for-go/file"
225+
"github.com/appwrite/sdk-for-go/id"
226+
"github.com/appwrite/sdk-for-go/storage"
227+
)
228+
229+
func main() {
230+
clt := client.New(
231+
client.WithEndpoint("https://<REGION>.cloud.appwrite.io/v1"),
232+
client.WithProject("<PROJECT_ID>"),
233+
client.WithKey("<API_KEY>"),
234+
)
235+
236+
service := storage.New(clt)
237+
238+
uploaded, err := service.CreateFile(
239+
"videos",
240+
id.Unique(),
241+
file.NewInputFile("./large-video.mp4", "large-video.mp4"),
242+
)
243+
_ = uploaded
244+
_ = err
245+
}
246+
```
247+
```server-rust
248+
use appwrite::Client;
249+
use appwrite::services::Storage;
250+
use appwrite::InputFile;
251+
use appwrite::id::ID;
252+
253+
#[tokio::main]
254+
async fn main() -> Result<(), Box<dyn std::error::Error>> {
255+
let client = Client::new()
256+
.set_endpoint("https://<REGION>.cloud.appwrite.io/v1")
257+
.set_project("<PROJECT_ID>")
258+
.set_key("<API_KEY>");
259+
260+
let storage = Storage::new(&client);
261+
262+
let file = InputFile::from_path("./large-video.mp4", None).await?;
263+
264+
let uploaded = storage.create_file(
265+
"videos",
266+
ID::unique(),
267+
file,
268+
None,
269+
).await?;
270+
271+
let _ = uploaded;
272+
Ok(())
273+
}
274+
```
275+
```server-kotlin
276+
import io.appwrite.Client
277+
import io.appwrite.ID
278+
import io.appwrite.models.InputFile
279+
import io.appwrite.services.Storage
280+
281+
suspend fun upload() {
282+
val client = Client()
283+
.setEndpoint("https://<REGION>.cloud.appwrite.io/v1")
284+
.setProject("<PROJECT_ID>")
285+
.setKey("<API_KEY>")
286+
287+
val storage = Storage(client)
288+
289+
val uploaded = storage.createFile(
290+
bucketId = "videos",
291+
fileId = ID.unique(),
292+
file = InputFile.fromPath("./large-video.mp4")
293+
)
294+
}
295+
```
296+
```server-swift
297+
import Appwrite
298+
299+
let client = Client()
300+
.setEndpoint("https://<REGION>.cloud.appwrite.io/v1")
301+
.setProject("<PROJECT_ID>")
302+
.setKey("<API_KEY>")
303+
304+
let storage = Storage(client)
305+
306+
let uploaded = try await storage.createFile(
307+
bucketId: "videos",
308+
fileId: ID.unique(),
309+
file: InputFile.fromPath("./large-video.mp4")
310+
)
311+
```
312+
```server-dotnet
313+
using Appwrite;
314+
using Appwrite.Models;
315+
using Appwrite.Services;
316+
317+
Client client = new Client()
318+
.SetEndPoint("https://<REGION>.cloud.appwrite.io/v1")
319+
.SetProject("<PROJECT_ID>")
320+
.SetKey("<API_KEY>");
321+
322+
Storage storage = new Storage(client);
323+
324+
File uploaded = await storage.CreateFile(
325+
bucketId: "videos",
326+
fileId: ID.Unique(),
327+
file: InputFile.FromPath("./large-video.mp4")
328+
);
329+
```
330+
{% /multicode %}
331+
332+
# Benchmarks
333+
334+
We benchmarked the Node SDK uploading files from 10 MB up to 1.28 GB, comparing the old sequential client against the new parallel client at the default concurrency of 8.
335+
336+
| File size | Sequential | Parallel | Speedup | Concurrency | Chunks |
337+
| --- | --- | --- | --- | --- | --- |
338+
| 10 MB | 2,650 ms | 2,472 ms | 1.07x | 1 | 2 |
339+
| 20 MB | 4,589 ms | 2,434 ms | 1.89x | 3 | 4 |
340+
| 40 MB | 9,413 ms | 2,680 ms | 3.51x | 7 | 8 |
341+
| 80 MB | 18,233 ms | 4,037 ms | 4.52x | 8 | 16 |
342+
| 160 MB | 36,606 ms | 6,545 ms | 5.59x | 8 | 32 |
343+
| 320 MB | 71,036 ms | 11,451 ms | 6.20x | 8 | 64 |
344+
| 640 MB | 141,923 ms | 20,863 ms | 6.80x | 8 | 128 |
345+
| 1.28 GB | 283,823 ms | 39,956 ms | **7.10x** | 8 | 256 |
346+
347+
The pattern is clear:
348+
349+
- **Small files** (a single chunk or two) cannot benefit from concurrency and stay close to the sequential baseline.
350+
- **Large files** have enough chunks to saturate the worker pool, so the speedup climbs steeply, reaching up to 7.10x at 1.28 GB.
351+
352+
Your mileage will vary with region, device, bucket location, and network, but the larger the file, the larger the win.
353+
354+
# Get started
355+
356+
Parallel chunked uploads are now available on Appwrite Cloud. Upgrade your Appwrite SDK to the latest release for your language and your existing `createFile` calls inherit the faster uploads automatically - no API migration, no extra configuration.
357+
358+
The Appwrite Console also uses parallel chunking, so file uploads through the Console are faster too.
359+
360+
# More resources
361+
362+
- [Appwrite Storage documentation](/docs/products/storage)
363+
- [The easiest way to add file uploads to your app](/blog/post/easiest-file-uploads)

‎src/routes/changelog/(entries)/2026-05-21-1.markdoc‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
---
2+
layout: changelog
3+
title: "Up to 7x faster Appwrite Storage uploads with parallel chunks"
4+
description: Appwrite SDKs now upload Storage chunks in parallel on runtimes with native concurrency. Our Node SDK benchmarks show up to 7.10x faster uploads on large files, with no API changes for developers.
5+
date: 2026-05-21
6+
cover: /images/blog/faster-storage-uploads-parallel-chunks/cover.avif
7+
---
8+
9+
Appwrite SDKs now upload **Storage** file chunks **in parallel** where the host runtime supports overlapping HTTP requests. Chunking, concurrency limits, and ordering are handled **inside the client**; your **`createFile`** calls stay the same.
10+
11+
In our Node SDK benchmarks, a 1.28 GB upload dropped from 4 minutes 44 seconds to under 40 seconds, up to a **7.10x** improvement at the default concurrency of 8. Smaller files see proportionally smaller gains since they have fewer chunks to overlap.
12+
13+
Available on Appwrite Cloud today.
14+
15+
{% arrow_link href="/blog/post/faster-storage-uploads-parallel-chunks" %}
16+
Read the announcement
17+
{% /arrow_link %}

0 commit comments

Comments
 (0)