Skip to content

Commit 441f60d

Browse files
r4victorhaydnli-shopify
authored andcommitted
Document File storage (dstackai#2755)
1 parent e4b43c0 commit 441f60d

2 files changed

Lines changed: 54 additions & 2 deletions

File tree

docs/docs/concepts/repos.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,8 @@ If the directory is cloned Git repo, [`dstack apply`](../reference/cli/dstack/ap
2828
If the directory is not a cloned Git repo, it uploads the entire directory.
2929

3030
Uploads are limited to 2MB. Use `.gitignore` to exclude unnecessary files from being uploaded.
31+
You can set the `DSTACK_SERVER_CODE_UPLOAD_LIMIT` environment variable to increase the default server limit.
32+
Increasing the limit is recommended only if you [configure an object storage](../guides/server-deployment.md).
3133

3234
### Initialize as a local directory
3335

docs/docs/guides/server-deployment.md

Lines changed: 52 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -162,7 +162,7 @@ For multi-replica server deployments, it's required to store logs externally.
162162
To store logs in AWS CloudWatch, set the `DSTACK_SERVER_CLOUDWATCH_LOG_GROUP` and
163163
the `DSTACK_SERVER_CLOUDWATCH_LOG_REGION` environment variables.
164164

165-
The log group must be created beforehand, `dstack` won't try to create it.
165+
The log group must be created beforehand. `dstack` won't try to create it.
166166

167167
??? info "Required permissions"
168168

@@ -218,6 +218,56 @@ To store logs using GCP Logging, set the `DSTACK_SERVER_GCP_LOGGING_PROJECT` env
218218

219219
</div>
220220

221+
## File storage
222+
223+
When using [repos](../concepts/repos.md), the `dstack` CLI uploads uncommitted local files and diffs to the server so that you can have access to them within runs. By default, the files are stored in the DB and each upload is limited to 2MB. You can configure an object storage to be used for uploads and increase the default limit by setting the `DSTACK_SERVER_CODE_UPLOAD_LIMIT` environment variable
224+
225+
### S3
226+
227+
To use S3 for storing uploaded files, set the `DSTACK_SERVER_S3_BUCKET` and `DSTACK_SERVER_BUCKET_REGION` environment variables.
228+
The bucket must be created beforehand. `dstack` won't try to create it.
229+
230+
??? info "Required permissions"
231+
232+
```json
233+
{
234+
"Version": "2012-10-17",
235+
"Statement": [
236+
{
237+
"Effect": "Allow",
238+
"Action": [
239+
"s3:GetObject",
240+
"s3:PutObject",
241+
"s3:DeleteObject",
242+
"s3:ListBucket"
243+
],
244+
"Resource": [
245+
"arn:aws:s3:::<bucket-name>",
246+
"arn:aws:s3:::<bucket-name>/*"
247+
]
248+
}
249+
]
250+
}
251+
```
252+
253+
### GCS
254+
255+
To use GCS for storing uploaded files, set the `DSTACK_SERVER_GCS_BUCKET` environment variable.
256+
The bucket must be created beforehand. `dstack` won't try to create it.
257+
258+
??? info "Required permissions"
259+
Ensure you've configured Application Default Credentials with the following permissions:
260+
261+
```
262+
storage.buckets.get
263+
storage.buckets.list
264+
storage.objects.get
265+
storage.objects.list
266+
storage.objects.create
267+
storage.objects.delete
268+
storage.objects.update
269+
```
270+
221271
## Encryption
222272

223273
By default, `dstack` stores data in plaintext. To enforce encryption, you
@@ -320,7 +370,7 @@ default_permissions:
320370
`dstack` follows the `{major}.{minor}.{patch}` versioning scheme.
321371
Backward compatibility is maintained based on these principles:
322372

323-
* The server backward compatibility is maintained across all minor and patch releases. The specific features can be removed but the removal is preceded with deprecation warnings for several minor releases. This means you can use older client versions with newer server versions.
373+
* The server backward compatibility is maintained on a best-effort basis across minor and patch releases. The specific features can be removed, but the removal is preceded with deprecation warnings for several minor releases. This means you can use older client versions with newer server versions.
324374
* The client backward compatibility is maintained across patch releases. A new minor release indicates that the release breaks client backward compatibility. This means you don't need to update the server when you update the client to a new patch release. Still, upgrading a client to a new minor version requires upgrading the server too.
325375

326376
## Server limits

0 commit comments

Comments
 (0)