Add Idempotency-Key header support for upload creation requests

[m7kvqbe1]

Summary

• Add support for the Idempotency-Key header on upload creation (POST) requests, allowing clients to safely retry without creating duplicate uploads
• Introduce an IdempotencyKeyStore interface with two implementations: file-based (for disk storage backends, persists across restarts) and in-memory (for cloud storage backends)
• Handle interrupted concatenation state: retry if offset is 0, return existing if complete, error if corrupted (0 < offset < size)
• Wire up the appropriate store automatically in the CLI based on the active storage backend

Motivation

Closes #1364

Upload creation requests (particularly concatenation) are not currently idempotent. When a concat request is interrupted (e.g. by a WAF timeout), the client never receives the response and retries, creating a duplicate final upload.

This implements the approach suggested by @Acconut in #1364: a general-purpose Idempotency-Key header that works for all upload creation requests, not just concatenation. For the filestore backend, key-to-upload-ID mappings are persisted to disk alongside upload data (as .idempotency-key files), so they survive server restarts.

See also #1365 for a previous concat-only approach.

Design

New interface (IdempotencyKeyStore) added to StoreComposer, following the same pattern as Locker:

type IdempotencyKeyStore interface {
    FindUploadID(ctx context.Context, key string) (string, error)
    StoreUploadID(ctx context.Context, key string, uploadID string) error
}

Two implementations shipped:

Backend	Store	Durability
filestore	fileidempotencystore (disk)	Survives restarts
S3/GCS/Azure	memoryidempotencystore (memory)	Within process lifetime

Handler logic (in PostFile):

1. Read Idempotency-Key header
2. If store configured and key present: look up existing upload
3. If found and complete: return existing (201 + Location)
4. If found and incomplete concat (offset==0): retry concatenation
5. If found and corrupted concat (0 < offset < size): return 500
6. If found and non-concat: return existing upload info for PATCH resume
7. If not found: create normally, store key->ID mapping

Opt-out: --disable-idempotency CLI flag.

Test plan

• TestIdempotency/ConcatRetryComplete -- retried concat with already-complete upload returns existing
• TestIdempotency/ConcatRetryIncomplete -- retried concat with offset==0 retries the concatenation
• TestIdempotency/ConcatRetryCorrupted -- retried concat with partial data returns error
• TestIdempotency/RegularUploadRetry -- retried regular upload returns existing with offset
• TestIdempotency/NoStoreConfigured -- header ignored when no store configured
• TestIdempotency/NoHeaderSent -- normal flow when no header
• TestIdempotency/DeletedUploadFallsThrough -- stale mapping to deleted upload creates new upload
• TestFileIdempotencyStore -- file-based store unit tests
• All existing handler tests continue to pass
• Full go test ./pkg/... passes

[Murderlon]

Review by Devin: https://app.devin.ai/review/tus/tusd/pull/1366

[m7kvqbe1]

Review by Devin: https://app.devin.ai/review/tus/tusd/pull/1366

Thanks @Murderlon.

I've added some fixups to this PR based on Devin's review:

• 684a8f8
• 6b918fa
• 917c98f

[Murderlon]

cc @Acconut

[Acconut]

Thank you for opening this PR. I'm wondering if this is actually capable of solving the timeout issues when concatenating large uploads. Tusd would just retry the concatenating operation when it detects an existing upload resource with the same idempotency key.

But if we assume that the concatenation operation times out because the underlying process is too slow, retrying the operation would likely not help. And tusd aborts the concatenation operation whenever the request times out on the client or proxy side. Unless the process is just sometimes too slow, and there is a chance that it will be faster at the new attempt, retrying offers little improvement. Tusd would actually have to resume the concatenation operation, but I'm not sure this is possible for all storage implementations. It might be doable for filestore and gcsstore, but not for s3store as far as I can tell.

Idempotency keys are a helpful tool to prevent duplicate upload resources from being created, but it might not be that helpful for solving concatenation failures on its own as I initially assumed. If we take a look at the original issue of duplicate uploads because of failed concatenations, the duplicates are just a symptom, but not the actual fault. That is the concatenation failing in the first place due to the request timing out. To prevent that the concatenation could be run in a background process, which has more time to complete than the short-lived request context. So even if the request times out after X seconds, the concatenation can continue. Deduplication via Idempotency-Key would still play an important role here as it ensures that client retrieves the original upload resources when it retries the upload concatenation request.

What do you think about this idea? The hurdle to getting this in tusd is higher since it currently doesn't have a system for running tasks in the background. But we might want to have this for other features as well (e.g. cleaning up old uploads).

[m7kvqbe1]

#1364 (comment)