You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/adr/002-large-file-upload-chunking.md
+32Lines changed: 32 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -234,6 +234,38 @@ Client requests {objName} → Swift reads manifest → Streams segments in order
234
234
235
235
**Implementation complexity:** High
236
236
237
+
### Alternative C: Swift Dynamic Large Objects (DLO) Instead of SLO
238
+
239
+
**Description:** Swift offers two large object mechanisms: Static Large Objects (SLO) and Dynamic Large Objects (DLO). Both are supported by the `ncw/swift` Go library used in this project. DLO uses a simpler manifest that only stores a segment prefix—Swift dynamically discovers segments matching that prefix at download time.
240
+
241
+
**Pros:**
242
+
- Simpler manifest structure (just a prefix, no segment list)
243
+
- Segments can be added or modified after manifest creation
244
+
- Slightly simpler upload logic (no need to track segment metadata)
245
+
246
+
**Cons:**
247
+
- **No integrity validation**: DLO manifests don't store segment ETags, so Swift cannot detect missing or corrupted segments. Downloads succeed with incomplete or wrong data rather than returning an error.
248
+
- **Race conditions**: Since segments are discovered dynamically, concurrent modifications can cause inconsistent reads (e.g., a download might see a partial set of segments if upload is in progress).
249
+
- **Unpredictable content**: The downloaded content depends on what segments exist at read time, not what was originally uploaded. If a segment is deleted (disk failure, bug, manual deletion), the file silently shrinks.
250
+
- **No size validation**: DLO cannot verify that the total size matches expectations.
251
+
252
+
**Why SLO is preferred:**
253
+
254
+
For a file storage system where data integrity matters, SLO provides critical guarantees that DLO lacks:
255
+
256
+
| Aspect | SLO | DLO |
257
+
|--------|-----|-----|
258
+
| Segment list | Explicit with ETags and sizes | Dynamic prefix matching |
259
+
| Integrity check | Swift validates each segment's ETag on download | None—trusts whatever exists |
| Modified segment | Detected—ETag mismatch causes failure | Silently returns different data |
262
+
| Consistency | Immutable after creation | Can change between reads |
263
+
| Client awareness | Failure is detectable (incomplete transfer) | No way to know data is missing |
264
+
265
+
**Note on UI chunked uploads:** DLO's flexibility (adding segments after manifest creation) might seem useful for resumable UI uploads, but it provides no real advantage. Both DLO and SLO support the same upload flow: client uploads segments, then server creates manifest on completion.
266
+
267
+
**Implementation complexity:** Low (similar to SLO), but unacceptable data integrity trade-offs.
0 commit comments