Skip to content

Commit 2b65191

Browse files
JuArceJuArce
authored
infra: update aggregation mode deployment (#2278)
1 parent 2596585 commit 2b65191

5 files changed

Lines changed: 111 additions & 25 deletions

File tree

‎.github/workflows/aggregation_mode.yml‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
name: "Start Aggregation Mode Server"
2+
3+
# Starts the Paperspace GPU server that runs the aggregation mode.
4+
#
5+
# The server is kept powered off to avoid 24/7 billing. This workflow boots it
6+
# once a day; on boot the machine runs `aggregation_mode.service`, which executes
7+
# the SP1 aggregations and then powers the machine off again
8+
# (see infra/aggregation_mode/run.sh).
9+
on:
10+
schedule:
11+
# 15:00 UTC == 12:00 GMT-3, every day. GitHub Actions cron is always in UTC.
12+
- cron: "0 15 * * *"
13+
workflow_dispatch:
14+
15+
jobs:
16+
start-server:
17+
name: Start Paperspace aggregation server
18+
runs-on: ubuntu-latest
19+
steps:
20+
- name: Start Paperspace machine
21+
env:
22+
PAPERSPACE_API_KEY: ${{ secrets.PAPERSPACE_API_KEY }}
23+
MACHINE_ID: ${{ secrets.PAPERSPACE_MACHINE_ID }}
24+
run: |
25+
set -euo pipefail
26+
27+
if [ -z "${PAPERSPACE_API_KEY}" ] || [ -z "${MACHINE_ID}" ]; then
28+
echo "::error::PAPERSPACE_API_KEY and PAPERSPACE_MACHINE_ID secrets must be set."
29+
exit 1
30+
fi
31+
32+
echo "Starting Paperspace machine ${MACHINE_ID}..."
33+
http_code=$(curl -sS -o response.json -w "%{http_code}" \
34+
-X PATCH "https://api.paperspace.com/v1/machines/${MACHINE_ID}/start" \
35+
-H "Authorization: Bearer ${PAPERSPACE_API_KEY}")
36+
37+
echo "Paperspace API responded with HTTP ${http_code}"
38+
cat response.json || true
39+
40+
# 2xx means the start request was accepted.
41+
if [ "${http_code}" -lt 200 ] || [ "${http_code}" -ge 300 ]; then
42+
echo "::error::Failed to start Paperspace machine (HTTP ${http_code})."
43+
exit 1
44+
fi
45+
46+
echo "Start request accepted. The machine will run the aggregation on boot and shut itself down afterwards."

‎infra/aggregation_mode/README.md‎

Lines changed: 23 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,33 @@
11
# Aggregation Mode Setup
22

3+
## Overview
4+
5+
The aggregation mode runs on a Paperspace GPU server. To avoid paying for the GPU
6+
24/7, the server is kept powered off and is only started once a day:
7+
8+
1. The [`Start Aggregation Mode Server`](../../.github/workflows/aggregation_mode.yml)
9+
GitHub Actions workflow runs daily at 15:00 UTC (12:00 GMT-3) and starts the
10+
Paperspace machine via the Paperspace API.
11+
2. On boot, `aggregation_mode.service` runs automatically and executes
12+
[`run.sh`](run.sh), which runs the SP1 and Risc0 aggregations.
13+
3. When the aggregations finish, `run.sh` powers the machine off again
14+
(`sudo shutdown -h now`), so Paperspace stops billing it.
15+
16+
The workflow needs:
17+
18+
- `PAPERSPACE_API_KEY` repository **secret** — a Paperspace API key.
19+
- `PAPERSPACE_MACHINE_ID` repository **secret** — the id of the GPU machine.
20+
321
## Setup on Server with GPU
422

523
To setup the server with GPU, follow the steps in [aggregation_mode.sh](aggregation_mode.sh).
624

7-
After running all the steps, `aggregation_mode.timer` will execute every 24hs the `aggregation_mode.service`
25+
After running all the steps, `aggregation_mode.service` is enabled to run on every
26+
boot. There is no timer anymore — the daily schedule is driven by the GitHub Actions
27+
workflow described above.
828

929
## Check Service Status
1030

11-
To check the status of the timer, run:
12-
13-
```bash
14-
systemctl status aggregation_mode.timer --user
15-
```
16-
1731
To check the status of the service, run:
1832

1933
```bash
@@ -22,7 +36,7 @@ systemctl status aggregation_mode.service --user
2236

2337
## Start Service manually
2438

25-
If you need to start the service manually, without waiting for the timer, run:
39+
If you need to start the service manually (the machine is already on), run:
2640

2741
```bash
2842
systemctl start aggregation_mode.service --user
@@ -33,7 +47,7 @@ systemctl start aggregation_mode.service --user
3347
To check the logs of the service, run:
3448

3549
```bash
36-
journalctl -xfeu aggregation_mode.service --user
50+
journalctl -xfeu aggregation_mode.service --user
3751
```
3852

3953
Note: You can add `-n <n_of_lines>` to limit the number of lines to show.

‎infra/aggregation_mode/aggregation_mode.service‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,4 +9,4 @@ ExecStartPre=sleep 60
99
ExecStart=/home/user/run.sh
1010

1111
[Install]
12-
WantedBy=multi-user.target
12+
WantedBy=default.target

‎infra/aggregation_mode/aggregation_mode.sh‎

Lines changed: 32 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,8 @@ done
2424
sudo loginctl enable-linger user
2525

2626
# Install other dependencies
27-
sudo apt install -y gcc pkg-config libssl-dev build-essential apt-transport-https ca-certificates curl software-properties-common nvtop
27+
sudo apt install -y gcc pkg-config libssl-dev build-essential apt-transport-https ca-certificates curl software-properties-common nvtop clang
28+
sudo apt-get install -y protobuf-compiler
2829

2930
# Install docker
3031
sudo apt-get update
@@ -52,14 +53,24 @@ sudo tailscale up --ssh --advertise-tags=tag:server && sudo tailscale set --auto
5253
# Install CUDA
5354
sudo add-apt-repository ppa:graphics-drivers/ppa
5455
sudo apt update
55-
sudo apt install nvidia-driver-570
56+
sudo apt install -y nvidia-driver-580
5657

57-
# If see errors
58-
sudo apt-mark unhold cuda-drivers cuda-toolkit-12-6 nvidia-dkms-565-server nvidia-fabricmanager-565 nvidia-headless-565-server nvidia-utils-565-server
59-
sudo apt update
60-
sudo apt install nvidia-driver-570
6158
sudo apt autoremove
62-
sudo apt autoclean
59+
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
60+
sudo dpkg -i cuda-keyring_1.1-1_all.deb
61+
sudo apt update
62+
sudo apt install -y cuda-toolkit-12-9
63+
## Add to ~/.bashrc
64+
export PATH=/usr/local/cuda-12.9/bin${PATH:+:${PATH}}
65+
export LD_LIBRARY_PATH=/usr/local/cuda-12.9/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
66+
67+
# If see errors
68+
#sudo apt-mark unhold cuda-drivers cuda-toolkit-12-6 nvidia-dkms-565-server nvidia-fabricmanager-565 nvidia-headless-565-server nvidia-utils-565-server
69+
#sudo apt update
70+
#sudo apt install nvidia-driver-570
71+
#sudo apt autoremove
72+
#sudo apt autoclean
73+
6374
sudo reboot
6475
nvidia-smi # To check if the driver is installed correctly
6576

@@ -108,26 +119,33 @@ read -p "Enter a block number for SP1 (last_aggregated_block): " num && echo "{\
108119
./infra/aggregation_mode/config_file.sh ./infra/aggregation_mode/config-proof-aggregator-risc0.template.yaml $HOME/config/config-proof-aggregator-risc0.yaml
109120
read -p "Enter a block number for Risc0 (last_aggregated_block): " num && echo "{\"last_aggregated_block\":$num}" > $HOME/config/proof-aggregator-risc0.last_aggregated_block.json
110121

122+
mkdir -p $HOME/repos/aggregation_mode/mainnet
123+
cd $HOME/repos/aggregation_mode/mainnet
124+
git clone https://github.com/yetanotherco/aligned_layer.git
125+
cd aligned_layer # Subsequent relative paths (cargo --path, ./infra/...) resolve from the repo root
111126
# Build the proof_aggregator
112-
make proof_aggregator_install
127+
#make proof_aggregator_install
128+
cargo install --path aggregation_mode/proof_aggregator --features prove,gpu --bin proof_aggregator_gpu --locked
129+
cargo install --path aggregation_mode/proof_aggregator --features prove --bin proof_aggregator_cpu --locked # This builds risc0 with CPU but sp1 is runnable with GPU
113130

114131
# Copy run script
115132
cp ./infra/aggregation_mode/run.sh $HOME/run.sh
116133
chmod 744 $HOME/run.sh
117134

118135
# Setup systemd service
136+
# The service is enabled to run on every boot (no timer). The machine is started
137+
# once a day by the GitHub Actions workflow (.github/workflows/aggregation_mode.yml),
138+
# runs the aggregation on startup and then shuts itself down (see run.sh).
119139
cp ./infra/aggregation_mode/aggregation_mode.service $HOME/.config/systemd/user/aggregation_mode.service
120-
cp ./infra/aggregation_mode/aggregation_mode.timer $HOME/.config/systemd/user/aggregation_mode.timer
121140

122-
#sudo systemctl enable aggregation_mode.service
123-
systemctl --user enable aggregation_mode.timer
124-
systemctl --user start aggregation_mode.timer
141+
systemctl --user daemon-reload
142+
systemctl --user enable aggregation_mode.service
125143

126144
# Run the proof_aggregator manually if you want
127145
systemctl --user start aggregation_mode.service
128146

129-
# Check timer status
130-
systemctl --user status aggregation_mode.timer
147+
# Check service status
148+
systemctl --user status aggregation_mode.service
131149

132150
# Check logs
133151
journalctl -xfeu aggregation_mode.service --user -n10

‎infra/aggregation_mode/run.sh‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
#!/bin/bash
22

3-
SLEEP=5
3+
SLEEP=60
44

55
echo "Starting Aggregation Mode in $SLEEP seconds..."
66
sleep $SLEEP
@@ -13,3 +13,11 @@ echo "SP1 Aggregation Mode finished"
1313
echo "Starting Risc0 Aggregation Mode..."
1414
AGGREGATOR=risc0 /home/user/.cargo/bin/proof_aggregator_gpu /home/user/config/config-proof-aggregator-risc0.yaml
1515
echo "Risc0 Aggregation Mode finished"
16+
17+
# Aggregation finished: power off the machine so Paperspace stops billing it.
18+
# The GitHub Actions workflow (.github/workflows/aggregation_mode.yml) starts the
19+
# machine again on the next scheduled run, and aggregation_mode.service runs this
20+
# script automatically on boot.
21+
echo "Aggregation finished, shutting down the machine in 1 minute..."
22+
sleep $SLEEP
23+
sudo shutdown -h now

0 commit comments

Comments
 (0)