Skip to content

Commit 8eab27e

Browse files
committed
docs: Added a DLO section to ADR
1 parent 6d3c3bd commit 8eab27e

1 file changed

Lines changed: 32 additions & 0 deletions

File tree

docs/adr/002-large-file-upload-chunking.md

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -234,6 +234,38 @@ Client requests {objName} → Swift reads manifest → Streams segments in order
234234
235235
**Implementation complexity:** High
236236
237+
### Alternative C: Swift Dynamic Large Objects (DLO) Instead of SLO
238+
239+
**Description:** Swift offers two large object mechanisms: Static Large Objects (SLO) and Dynamic Large Objects (DLO). Both are supported by the `ncw/swift` Go library used in this project. DLO uses a simpler manifest that only stores a segment prefix—Swift dynamically discovers segments matching that prefix at download time.
240+
241+
**Pros:**
242+
- Simpler manifest structure (just a prefix, no segment list)
243+
- Segments can be added or modified after manifest creation
244+
- Slightly simpler upload logic (no need to track segment metadata)
245+
246+
**Cons:**
247+
- **No integrity validation**: DLO manifests don't store segment ETags, so Swift cannot detect missing or corrupted segments. Downloads succeed with incomplete or wrong data rather than returning an error.
248+
- **Race conditions**: Since segments are discovered dynamically, concurrent modifications can cause inconsistent reads (e.g., a download might see a partial set of segments if upload is in progress).
249+
- **Unpredictable content**: The downloaded content depends on what segments exist at read time, not what was originally uploaded. If a segment is deleted (disk failure, bug, manual deletion), the file silently shrinks.
250+
- **No size validation**: DLO cannot verify that the total size matches expectations.
251+
252+
**Why SLO is preferred:**
253+
254+
For a file storage system where data integrity matters, SLO provides critical guarantees that DLO lacks:
255+
256+
| Aspect | SLO | DLO |
257+
|--------|-----|-----|
258+
| Segment list | Explicit with ETags and sizes | Dynamic prefix matching |
259+
| Integrity check | Swift validates each segment's ETag on download | None—trusts whatever exists |
260+
| Missing segment | Detected—connection dropped, client receives partial results | **Silently succeeds**—"happily ignores" the missing segment |
261+
| Modified segment | Detected—ETag mismatch causes failure | Silently returns different data |
262+
| Consistency | Immutable after creation | Can change between reads |
263+
| Client awareness | Failure is detectable (incomplete transfer) | No way to know data is missing |
264+
265+
**Note on UI chunked uploads:** DLO's flexibility (adding segments after manifest creation) might seem useful for resumable UI uploads, but it provides no real advantage. Both DLO and SLO support the same upload flow: client uploads segments, then server creates manifest on completion.
266+
267+
**Implementation complexity:** Low (similar to SLO), but unacceptable data integrity trade-offs.
268+
237269
## Decision
238270
239271
**Recommended approach: Storage-Level Chunking**

0 commit comments

Comments
 (0)