Skip to content

TOOLFORGE.md

Latest commit

 

History

History
232 lines (159 loc) · 6.36 KB

TOOLFORGE.md

File metadata and controls

232 lines (159 loc) · 6.36 KB
Raw

Toolforge Deployment

Toolforge is Wikimedia's hosting platform for community tools. This guide walks you through deploying Scribe-Server on Toolforge from scratch — no prior Toolforge experience needed. To deploy Scribe Server here, you first apply at toolsadmin.wikimedia.org with the relevant project details. After your application is approved, you gain SSH access and can set up the environment, database, and web service described below.

Contents

Practical Workflow

SSH into Toolforge

Connect to the login node using your Wikimedia developer account:

ssh {user_id}@login.toolforge.org

Once inside, switch to the Scribe tool account so all subsequent commands run in the correct project context:

become testserver-scribe

Then clone the repository into your project directory:

git clone https://github.com/scribe-org/Scribe-Server.git
cd Scribe-Server

Back to top.

Set Up Go

Toolforge's pre-built web images do not include Go, so you install it manually into your home directory.

Note: go-sqlite3 requires CGo and make, neither of which is available on Toolforge. Use a pure-Go SQLite driver instead.

# Download Go
wget https://go.dev/dl/go1.23.6.linux-amd64.tar.gz

# Extract the tarball
tar -xzf go1.23.6.linux-amd64.tar.gz -C ~/

# Rename the directory for organization
mv ~/go ~/go1.23

# Persist environment variables across sessions
echo 'export GOROOT=$HOME/go1.23' >> ~/.bashrc
echo 'export PATH=$PATH:$GOROOT/bin' >> ~/.bashrc

# Apply changes to the current shell
source ~/.bashrc

# Verify the installation
go version

# Clean up the downloaded archive
rm go1.23.6.linux-amd64.tar.gz

Why this layout:

  • Go is extracted to ~/go1.23 rather than ~/go to make the version explicit and avoid conflicts if you later upgrade.
  • GOROOT must point at your custom location because the system has no Go in PATH.
  • Running go run . inside Toolforge binds to 0.0.0.0:8000 — this is expected behavior within the Toolforge network.

Back to top.

Configure Database

Toolforge provides a shared MariaDB cluster. Your credentials are pre-written to ~/replica.my.cnf during tool creation.

First, read your credentials:

cat ~/replica.my.cnf

Then copy the example config and fill in the values:

mv config-example.yaml config.yaml
nano config.yaml
# Server configuration
hostPort: 8000
fileSystem: "./packs"

# Database configuration
database:
  user: {user}
  password: {password}
  host: tools-db.tools.eqiad1.wikimedia.cloud
  port: "3306"
  name: {user}__scribe_server_p

Replace {user} and {password} with the values from replica.my.cnf. The database name follows the Toolforge convention: {user}__<db_name>.

To inspect the database directly at any time:

mysql --defaults-file=~/replica.my.cnf \
  -h tools-db.tools.eqiad1.wikimedia.cloud \
  {user}__scribe_server_p

Example:

mysql --defaults-file=~/replica.my.cnf \
  -h tools-db.tools.eqiad1.wikimedia.cloud \
  s123456__scribe_server_p

Back to top.

Create Start Script

Note

The following only needs to be ran once.

~/Scribe-Server/start-script.sh << 'EOF'
#!/bin/bash
cd /data/project/testserver-scribe/Scribe-Server
export PORT=8000
./Scribe-Server
EOF

Make the script executable:

chmod +x ~/Scribe-Server/start-script.sh

Back to top.

Build and Run the Server

Each time you deploy an update, stop the running service, pull the latest code, rebuild the binary, and restart:

chmod +x start-script.sh
toolforge webservice stop
git pull origin main
go build -o Scribe-Server .
toolforge webservice --mem 4Gi --cpu 2 jdk17 start \
  /data/project/testserver-scribe/Scribe-Server/start-script.sh

kubectl --namespace=tool-scribe-server get ingress  # check for URL
kubectl logs -l toolforge=tool --tail=50 # see logs; last 50

Why this sequence:

  • toolforge webservice stop ensures the old binary is not locked when you overwrite it.
  • go build -o Scribe-Server . produces a statically-linked binary that Toolforge can execute directly.
  • The --mem 4Gi --cpu 2 flags allocate enough headroom for data loading on startup.

Back to top.

Python Tooling Setup

Note

The following is needed to Run Scribe-Data Fetch Script Successfully.

If you need Python tooling in the project, open a Python 3.13 shell and bootstrap pip:

toolforge webservice --mem 4Gi python3.13 shell

python3 -m venv .venv

source venv/bin/activate

curl -sS https://bootstrap.pypa.io/get-pip.py | python

Then run the following in the root of the project:

./update_data.sh true # we pass `true` to skip DB migration

Once done, exit the python shell by running: exit

Then run the migration:

go build -o ./bin/migrate-scribe-data ./cmd/migrate # to build migration file
./bin/migrate  # to run the migration

Back to top.

Install PyICU

Note

The following should be done if ICU Detection Fails.

The standard PyICU build uses pkg-config or icu-config to locate ICU headers and libraries. Neither tool is installed on Toolforge, so you must set the paths manually before running pip install:

export ICU_VERSION=76
export PYICU_LFLAGS="-L/usr/lib/x86_64-linux-gnu -licui18n -licuuc -licudata"
export PYICU_CFLAGS="-I/usr/include"
pip install PyICU

Why these variables:

  • ICU_VERSION tells the build script which header subdirectory to target.
  • PYICU_LFLAGS points the linker at the system ICU shared objects already present on Toolforge nodes.
  • PYICU_CFLAGS points the compiler at the system ICU headers.

Back to top.