diff --git a/src/routes/docs/products/storage/+layout.svelte b/src/routes/docs/products/storage/+layout.svelte index 122404835c..dd84bb33e9 100644 --- a/src/routes/docs/products/storage/+layout.svelte +++ b/src/routes/docs/products/storage/+layout.svelte @@ -48,6 +48,10 @@ { label: 'Image transformations', href: '/docs/products/storage/images' + }, + { + label: 'S3 API', + href: '/docs/products/storage/s3' } ] }, diff --git a/src/routes/docs/products/storage/s3/+page.markdoc b/src/routes/docs/products/storage/s3/+page.markdoc new file mode 100644 index 0000000000..33f185f8a4 --- /dev/null +++ b/src/routes/docs/products/storage/s3/+page.markdoc @@ -0,0 +1,240 @@ +--- +layout: article +title: S3 API +description: Connect any S3-compatible client, SDK, or tool to Appwrite Storage. Configure credentials once, then manage buckets, objects, and multipart uploads over the S3 API. +--- + +Appwrite Storage exposes an S3-compatible API, so you can point the AWS CLI, the AWS SDKs, and third-party tools like rclone or s3cmd at your buckets and files. The API speaks AWS Signature Version 4 and maps standard S3 operations onto Appwrite Storage, so most existing S3 code works after you change three settings: the endpoint, the credentials, and the region. + +The S3 API is available on Appwrite Cloud. It needs a project ID and an [API key](/docs/advanced/security/api-keys) with the storage scopes described below, and it works alongside your existing data: buckets and files are addressable over both the native Storage API and the S3 API at the same time, with no migration required. + +# Connect an S3 client {% #connect-an-s3-client %} + +Configure your client with the following values. S3 clients authenticate with an access key ID and a secret access key, which Appwrite maps to your project ID and an API key secret. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://.cloud.appwrite.io/v1/s3` | +| Access key ID | Your Appwrite project ID | +| Secret access key | An Appwrite API key secret | +| Region | `auto` or `us-east-1` | +| Signature version | AWS Signature Version 4 (SigV4) | +| Addressing style | Path-style only | + +The `` in the endpoint is your Appwrite Cloud region (for example, `fra` or `nyc`), the same host you use for the rest of the Appwrite API. The S3 `region` setting is separate, and Appwrite ignores it. Set it to `auto` or `us-east-1` and leave it alone. Some S3 clients require a region to sign requests, but the value has no effect. + +# Configure your client {% #configure-your-client %} + +Point your client at the endpoint from the table above, set the region to `auto`, enable path-style addressing, and pass your project ID and API key secret as the credentials. + +{% multicode %} +```bash +# AWS CLI +aws configure set aws_access_key_id +aws configure set aws_secret_access_key +aws configure set region auto + +# Appwrite serves path-style URLs only, so force path addressing +aws configure set default.s3.addressing_style path + +# Pass the Appwrite endpoint on every command +aws s3 ls --endpoint-url https://.cloud.appwrite.io/v1/s3 +``` + +```js +// Node.js: @aws-sdk/client-s3 (v3) +import { S3Client } from '@aws-sdk/client-s3'; + +const client = new S3Client({ + endpoint: 'https://.cloud.appwrite.io/v1/s3', + region: 'auto', + forcePathStyle: true, + credentials: { + accessKeyId: '', + secretAccessKey: '' + } +}); +``` + +```python +# Python: boto3 +import boto3 +from botocore.config import Config + +s3 = boto3.client( + 's3', + endpoint_url='https://.cloud.appwrite.io/v1/s3', + region_name='auto', + aws_access_key_id='', + aws_secret_access_key='', + config=Config(signature_version='s3v4', s3={'addressing_style': 'path'}) +) +``` + +```go +// Go: aws-sdk-go-v2 +cfg, err := config.LoadDefaultConfig(context.TODO(), + config.WithRegion("auto"), + config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider( + "", "", "", + )), +) +if err != nil { + log.Fatal(err) +} + +client := s3.NewFromConfig(cfg, func(o *s3.Options) { + o.BaseEndpoint = aws.String("https://.cloud.appwrite.io/v1/s3") + o.UsePathStyle = true +}) +``` + +```php +// PHP: aws/aws-sdk-php +use Aws\S3\S3Client; + +$client = new S3Client([ + 'version' => 'latest', + 'region' => 'auto', + 'endpoint' => 'https://.cloud.appwrite.io/v1/s3', + 'use_path_style_endpoint' => true, + 'credentials' => [ + 'key' => '', + 'secret' => '', + ], +]); +``` +{% /multicode %} + +# API key scopes {% #api-key-scopes %} + +Create an API key in your project's **Overview** settings, then grant the storage scopes your workload needs. The project ID identifies your project, while the API key secret both authenticates the request and determines what it is allowed to do. + +| Access level | Required scopes | +| --- | --- | +| Read-only (list and download) | `buckets.read`, `files.read` | +| Write (create buckets, upload, delete) | `buckets.write`, `files.write` | +| Copy (`CopyObject`, `UploadPartCopy`) | `files.read` and `files.write` | +| Full read and write | `buckets.read`, `files.read`, `buckets.write`, `files.write` | + +Copying an object reads the source and writes the destination, so `CopyObject` and `UploadPartCopy` need both `files.read` and `files.write`. A request signed with a secret that does not match a project API key, or a key missing the required scope, is rejected with `AccessDenied` (HTTP 403). + +# Object key mapping {% #object-key-mapping %} + +S3 buckets map directly to Appwrite Storage buckets, and S3 objects map to files. Creating a bucket over S3 creates an Appwrite bucket with the bucket name set to the ID you provided and Appwrite's default settings. Any bucket you already have in your project is usable over S3 immediately. + +Every object maps to a file. When you upload an object, Appwrite stores it as a file with an automatically generated file ID and uses the object key you provided as the file's name. The **canonical key**, the key returned by list operations and expected by read, copy, and delete operations, is: + +```text +{fileId}-{fileName} +``` + +For example, uploading to the key `reports/january.pdf` creates a file whose name is `reports/january.pdf`, and the object is then listed under a canonical key such as `6710e2...f3-reports/january.pdf`. + +Listing a bucket to discover the `{fileId}-{fileName}` key is also how the S3 API interoperates with files created through the native Storage API and Console: list the bucket, then address the file over S3 with its canonical key. + +{% info title="Uploads never overwrite by key" %} +Because a new file ID is generated on every plain upload, uploading twice with the same key creates two distinct objects instead of overwriting. To read, copy, overwrite, or delete an object, use its canonical key from a list operation. Uploading to a canonical key that already exists overwrites that object in place. +{% /info %} + +# Example commands {% #example-commands %} + +Once your client is configured, everyday S3 commands work as usual. These AWS CLI examples assume you have run `aws configure` as shown above and set an alias or environment default for the endpoint. + +```bash +# Create a bucket +aws s3api create-bucket --bucket my-bucket \ + --endpoint-url https://.cloud.appwrite.io/v1/s3 + +# Upload a file (uses multipart automatically for large files) +aws s3 cp ./january.pdf s3://my-bucket/reports/january.pdf \ + --endpoint-url https://.cloud.appwrite.io/v1/s3 + +# List objects to discover canonical keys +aws s3 ls s3://my-bucket --recursive \ + --endpoint-url https://.cloud.appwrite.io/v1/s3 + +# Download using the canonical {fileId}-{fileName} key from the listing +aws s3api get-object --bucket my-bucket \ + --key 6710e2f3-reports/january.pdf ./january.pdf \ + --endpoint-url https://.cloud.appwrite.io/v1/s3 +``` + +# Bucket operations {% #bucket-operations %} + +All operations are addressed with path-style URLs under `/v1/s3`. The following bucket operations are supported. + +| Operation | Description | +| --- | --- | +| `ListBuckets` | List the project's buckets. | +| `CreateBucket` | Create a bucket. The bucket ID becomes the bucket name, with Appwrite's default settings. | +| `HeadBucket` | Check that a bucket exists and is accessible. | +| `DeleteBucket` | Delete an empty bucket. Deleting a bucket that still contains objects returns `BucketNotEmpty` (HTTP 409). | +| `GetBucketLocation` | Returns `us-east-1`. | +| `GetBucketAcl`, `PutBucketAcl` | Compatibility responses only. See [Limitations](#limitations). | + +# Object operations {% #object-operations %} + +The following object operations are supported. + +| Operation | Description | +| --- | --- | +| `PutObject` | Upload an object. Returns an `ETag`. Honors `Content-Type`, `x-amz-meta-*` user metadata, and `x-amz-server-side-encryption`. | +| `GetObject` | Download an object. Supports `Range` requests (HTTP 206), and the `If-None-Match` (HTTP 304) and `If-Match` (HTTP 412) conditional headers. | +| `HeadObject` | Retrieve an object's metadata (size, content type, `ETag`, user metadata) without the body. | +| `CopyObject` | Copy an object using the `x-amz-copy-source` header. Supports `x-amz-metadata-directive: REPLACE` to replace metadata. | +| `DeleteObject` | Delete a single object. | +| `DeleteObjects` | Delete multiple objects in one request, including quiet mode. | +| `ListObjects` | List objects in a bucket, with `prefix` filtering. | +| `ListObjectsV2` | List objects with `prefix`, `delimiter`, `max-keys`, `continuation-token`, and `start-after`. | +| `GetObjectAcl`, `PutObjectAcl` | Compatibility responses only. See [Limitations](#limitations). | + +Content type is taken from the `Content-Type` you send. When that is missing or generic (`application/octet-stream`), Appwrite infers a type from the file extension. Buckets keep their constraints under the S3 API: uploads that exceed the bucket's maximum file size or use a disallowed extension are rejected, and disabled buckets are treated as not found. + +# Multipart uploads {% #multipart-uploads %} + +Large files are uploaded in parts. Standard SDK multipart helpers, such as the AWS CLI's `aws s3 cp` and the SDK transfer managers, use these operations automatically. + +| Operation | Description | +| --- | --- | +| `CreateMultipartUpload` | Begin a multipart upload and receive an `UploadId`. | +| `UploadPart` | Upload a single part. Returns the part's `ETag`. | +| `UploadPartCopy` | Upload a part by copying a byte range from another object, using `x-amz-copy-source` and `x-amz-copy-source-range`. | +| `CompleteMultipartUpload` | Assemble the uploaded parts into the final object. | +| `AbortMultipartUpload` | Discard an in-progress multipart upload and its parts. | +| `ListMultipartUploads` | List in-progress multipart uploads in a bucket. | +| `ListParts` | List the parts already uploaded for an `UploadId`. | + +{% info title="Multipart requirements" %} +Parts must be contiguous and start at part number 1. Completing an upload with non-contiguous part numbers (for example parts 1, 5, and 100) is rejected. Sequential uploads, which is what SDK multipart flows produce, are unaffected. In-progress uploads are kept for 24 hours before they are cleaned up, so complete or abort within that window. +{% /info %} + +# Limitations {% #limitations %} + +The S3 API targets the operations most clients depend on. Keep the following in mind. + +- **Path-style addressing only.** Virtual-hosted-style URLs (`https://.host/...`) are not supported. Enable path-style addressing in your client, as shown in [Configure your client](#configure-your-client). +- **Signature Version 4 only.** Requests are authenticated with SigV4, and the request timestamp must be within 15 minutes of the server's clock. +- **Reading back objects.** Reads, copies, overwrites, and deletes address objects by their canonical `{fileId}-{fileName}` key from a list operation, not by the raw key you uploaded with. See [Object key mapping](#object-key-mapping). +- **ACLs are compatibility responses.** `GetObjectAcl`, `PutObjectAcl`, `GetBucketAcl`, and `PutBucketAcl` return a canned private ACL and are accepted for tooling compatibility. They do not change access. Manage access with [Appwrite permissions](/docs/products/storage/permissions) and API key scopes instead. +- **Folders are not supported.** Uploading a folder marker (a key ending in `/`) returns `NotImplemented` (HTTP 501). Object keys can still contain `/`, so prefixes behave as expected. +- **Object listings are built in memory.** `ListObjects` and `ListObjectsV2` load the matching objects from the database and order them in the API. `ListObjects` returns every matching object in a single response, while `ListObjectsV2` honors `max-keys` (capped at 1000) and paginates with `IsTruncated` and a `NextContinuationToken`. No objects are silently dropped, but listing very large buckets increases per-request latency and memory. +- **Custom file IDs with hyphens.** The canonical key is split on the first `-`. Objects created through the S3 API use generated IDs without hyphens, so this only affects interop with pre-existing files whose custom ID contains a `-`. A mismatch returns `NoSuchKey` rather than the wrong object. +- **Unsupported S3 features.** Object tagging, CORS, lifecycle, bucket policies and policy status, encryption configuration, ownership controls, notifications, and versioning are not supported. Requests to these sub-resources return `NotImplemented` (HTTP 501). + +# Error responses {% #error-responses %} + +Errors are returned as standard S3 XML error documents with an S3 error code and HTTP status. + +| S3 error code | HTTP status | Meaning | +| --- | --- | --- | +| `NoSuchBucket` | 404 | The bucket does not exist, is disabled, or is not accessible. | +| `NoSuchKey` | 404 | The object key does not exist. | +| `BucketNotEmpty` | 409 | The bucket still contains objects and cannot be deleted. | +| `BucketAlreadyOwnedByYou` | 409 | A bucket with that ID already exists in your project. | +| `AccessDenied` | 403 | The signature is invalid, or the API key is expired or missing a required scope. | +| `InvalidRange` | 416 | The requested `Range` cannot be satisfied. | +| `PreconditionFailed` | 412 | An `If-Match` precondition did not hold. | +| `NotImplemented` | 501 | The requested S3 feature is not supported. | +| `InvalidRequest` | 400 | The request was malformed or violated a bucket constraint. | +| `InternalError` | 500 | An unexpected server error occurred. |