Skip to content

Commit 4b4e731

Browse files
peterschmidt85peterschmidt85
committed
Updated runner and shim contributing guide #2534 (review feedback)
1 parent 2789f35 commit 4b4e731

3 files changed

Lines changed: 64 additions & 46 deletions

File tree

‎.justfile‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Root justfile
2+
#
3+
# This justfile serves as the main entry point to recipes from different components.
4+
#
5+
# Run `just` to see all available commands.
6+
#
7+
# Components:
8+
# * runner/justfile – Building and uploading dstack runner and shim
9+
10+
default:
11+
@just --list
12+
13+
set allow-duplicate-recipes
14+
15+
import "runner/.justfile"
16+
17+
# TODO: Add frontend/justfile for managing frontend development tasks

justfile renamed to runner/.justfile

Lines changed: 39 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -1,61 +1,58 @@
11
# Justfile for building and uploading dstack runner and shim
22
#
3+
# Run `just` to see all available commands
4+
#
5+
# IMPORTANT: Change the s3_bucket and version variables below before running any commands.
6+
#
37
# Build Process:
48
# - Runner is always built for linux/amd64
59
# - Shim can be built for any platform (defaults to host platform)
610
# - When uploading, shim is automatically built for linux/amd64
711
#
8-
# Upload Process:
9-
# - Both runner and shim are uploaded to S3 with version prefix
10-
# - Shim is validated to ensure it's built for linux/amd64 before upload
11-
# - Upload commands automatically rebuild binaries before uploading
12-
#
1312
# Development Workflows:
1413
# - Local Development:
1514
# * Use build recipes to build binaries for local testing
16-
# * See runner/README.md for instructions on running dstack server with local binaries
15+
# * See README.md for instructions on running dstack server with local binaries
1716
# * No need to upload binaries for local development
1817
#
1918
# - Remote Development:
2019
# * Use upload recipes to build and upload binaries to S3
21-
# * See runner/README.md for instructions on running dstack server with uploaded binaries
20+
# * See README.md for instructions on running dstack server with uploaded binaries
2221
# * Upload is required for testing with standard backends (including SSH fleets)
23-
#
24-
# Common Commands:
25-
# - just build: Build both runner and shim
26-
# - just upload: Upload both runner and shim
27-
# - just build-upload: Build and upload both
28-
# - just upload-shim: Build and upload only shim
29-
# - just upload-runner: Build and upload only runner
3022

31-
# Default recipe to display available commands
3223
default:
3324
@just --list
3425

3526
# Export version variable
27+
# Must be changed to a unique value to avoid conflicts with other developers using the same bucket
3628
export version := "0.0.0"
3729

30+
# S3 bucket name - must be set to a bucket that the developer has access to
31+
export s3_bucket := "dstack-runner-downloads-stgn"
32+
33+
# Download URLs
34+
export runner_download_url := "s3://" + s3_bucket + "/" + version + "/binaries/dstack-runner-linux-amd64"
35+
export shim_download_url := "s3://" + s3_bucket + "/" + version + "/binaries/dstack-shim-linux-amd64"
36+
3837
# Shim build configuration
3938
export shim_os := ""
4039
export shim_arch := ""
4140

42-
# Download URLs
43-
export runner_download_url := "s3://dstack-runner-downloads-stgn/" + version + "/binaries/dstack-runner-linux-amd64"
44-
export shim_download_url := "s3://dstack-runner-downloads-stgn/" + version + "/binaries/dstack-shim-linux-amd64"
45-
4641
# Build runner
47-
build-runner:
42+
[private]
43+
build-runner-binary:
4844
#!/usr/bin/env bash
4945
set -e
5046
echo "Building runner for linux/amd64"
51-
cd runner/cmd/runner && GOOS=linux GOARCH=amd64 go build
47+
cd {{source_directory()}}/cmd/runner && GOOS=linux GOARCH=amd64 go build
5248
echo "Runner build complete!"
5349

5450
# Build shim
55-
build-shim:
51+
[private]
52+
build-shim-binary:
5653
#!/usr/bin/env bash
5754
set -e
58-
cd runner/cmd/shim
55+
cd {{source_directory()}}/cmd/shim
5956
if [ -n "$shim_os" ] && [ -n "$shim_arch" ]; then
6057
echo "Building shim for $shim_os/$shim_arch"
6158
GOOS=$shim_os GOARCH=$shim_arch go build
@@ -66,47 +63,47 @@ build-shim:
6663
echo "Shim build complete!"
6764

6865
# Build both runner and shim
69-
build: build-runner build-shim
66+
build-runner: build-runner-binary build-shim-binary
7067
echo "Build complete! Linux AMD64 binaries are in their respective cmd directories."
7168

7269
# Clean build artifacts
73-
clean:
74-
rm -f runner/cmd/runner/runner
75-
rm -f runner/cmd/shim/shim
70+
clean-runner:
71+
rm -f {{source_directory()}}/cmd/runner/runner
72+
rm -f {{source_directory()}}/cmd/shim/shim
7673
echo "Build artifacts cleaned!"
7774

7875
# Run tests for runner and shim
79-
test:
80-
cd runner && go test -v ./...
76+
test-runner:
77+
cd {{source_directory()}} && go test -v ./...
8178

8279
# Validate shim is built for linux/amd64
83-
validate-shim:
80+
[private]
81+
validate-shim-binary:
8482
#!/usr/bin/env bash
8583
set -e
86-
if ! file runner/cmd/shim/shim | grep -q "ELF 64-bit LSB executable, x86-64"; then
84+
if ! file {{source_directory()}}/cmd/shim/shim | grep -q "ELF 64-bit LSB executable, x86-64"; then
8785
echo "Error: Shim must be built for linux/amd64 for upload"
8886
exit 1
8987
fi
9088

9189
# Upload both runner and shim to S3
92-
upload: upload-runner upload-shim
90+
upload-runner: upload-runner-binary upload-shim-binary
9391

9492
# Upload runner to S3
95-
upload-runner:
93+
[private]
94+
upload-runner-binary:
9695
#!/usr/bin/env bash
9796
set -e
98-
just build-runner
99-
aws s3 cp runner/cmd/runner/runner "{{runner_download_url}}" --acl public-read
97+
just build-runner-binary
98+
aws s3 cp {{source_directory()}}/cmd/runner/runner "{{runner_download_url}}" --acl public-read
10099
echo "Uploaded runner to S3"
101100

102101
# Upload shim to S3
103-
upload-shim:
102+
[private]
103+
upload-shim-binary:
104104
#!/usr/bin/env bash
105105
set -e
106-
just --set shim_os linux --set shim_arch amd64 build-shim
107-
just validate-shim
108-
aws s3 cp runner/cmd/shim/shim "{{shim_download_url}}" --acl public-read
106+
just --set shim_os linux --set shim_arch amd64 build-shim-binary
107+
just validate-shim-binary
108+
aws s3 cp {{source_directory()}}/cmd/shim/shim "{{shim_download_url}}" --acl public-read
109109
echo "Uploaded shim to S3"
110-
111-
# Build and upload both runner and shim to S3
112-
build-upload: build upload

‎runner/README.md‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -85,18 +85,22 @@ You can also test the built shim and runner using standard backends (including S
8585
> [!NOTE]
8686
> To run with standard backends, both the runner and shim must be built for linux/amd64.
8787

88-
Build the runner and shim, and upload them to S3 automatically using `just` (see [justfile](../justfile)).
88+
Build the runner and shim, and upload them to S3 automatically using `just` (see [`justfile`](justfile)).
89+
90+
> [!IMPORTANT]
91+
> Before running any `just` commands that upload to S3, make sure to modify the `s3_bucket` and `version`
92+
> variables in the [`justfile`](justfile).
8993

9094
```shell
9195
just upload
9296
```
9397

94-
To use the built shim and runner with the dstack server, pass the URLs via `DSTACK_SHIM_DOWNLOAD_URL` and `DSTACK_RUNNER_DOWNLOAD_URL`:
98+
To use the built shim and runner with the dstack server, pass the URLs via `DSTACK_SHIM_DOWNLOAD_URL` and `DSTACK_RUNNER_DOWNLOAD_URL`. Make sure to use the same `s3_bucket` and `version` values that you set in the [`justfile`](justfile).
9599

96100
```shell
97-
DSTACK_SHIM_DOWNLOAD_URL=https://dstack-runner-downloads-stgn.s3.eu-west-1.amazonaws.com/0.0.0/binaries/dstack-shim-linux-amd64
101+
DSTACK_SHIM_DOWNLOAD_URL=https://<s3_bucket>.s3.<s3_bucket_region>.amazonaws.com/<version>/binaries/dstack-shim-linux-amd64
98102
99-
DSTACK_RUNNER_DOWNLOAD_URL=https://dstack-runner-downloads-stgn.s3.eu-west-1.amazonaws.com/0.0.0/binaries/dstack-runner-linux-amd64
103+
DSTACK_RUNNER_DOWNLOAD_URL=https://<s3_bucket>.s3.<s3_bucket_region>.amazonaws.com/<version>/binaries/dstack-runner-linux-amd64
100104
101105
dstack server --log-level=debug
102106
```

0 commit comments

Comments
 (0)