Skip to content

DEPLOY.md

Latest commit

 

History

History
154 lines (111 loc) · 7.34 KB

DEPLOY.md

File metadata and controls

154 lines (111 loc) · 7.34 KB
Raw

Deployment Guide

This guide outlines the steps to deploy this project using Modal.

Pre-requisites:

  • A Modal account.
  • The modal CLI tool installed.

Steps:

  1. Create a GitHub Personal Access Token (PAT):

    ⚠️ Security Best Practice: Use a Fine-grained Personal Access Token instead of a classic PAT for minimal permissions.

    Option A: Fine-grained PAT (Recommended)

    • Go to GitHub Settings → Developer settings → Personal access tokens → Fine-grained tokens
    • Set Repository access to "Only select repositories" and choose your target repos
    • Under Permissions, set:
      • Actions: Read and Write (required for JIT runner registration)
      • Administration: Read and Write (required for runner management)
    • This limits the token to specific repositories only

    Option B: Classic PAT (Less Secure)

    • Generate a PAT with the repo and workflow scope
    • ⚠️ This grants broad access to all your repositories

  2. Define a Webhook Secret (Mandatory):

    • Create a random string to use as your WEBHOOK_SECRET. This is required for validating that requests actually come from GitHub.
    • Generate a secure random secret: 
      openssl rand -hex 32

  3. Configure Repository Allowlist (Recommended):

    • For additional security, specify which repositories can trigger runners:
    • Format: comma-separated list of owner/repo names

  4. Create a Modal Secret:

    modal secret create github-full-secret \
  GITHUB_TOKEN=your_pat_here \
  WEBHOOK_SECRET=your_webhook_secret_here \
  ALLOWED_REPOS="owner/repo1,owner/repo2"
    • Replace your_pat_here with the PAT you generated.
    • Replace your_webhook_secret_here with your random string.
    • Replace owner/repo1,owner/repo2 with your allowed repositories (or omit to allow all).

    Optional Configuration:

    # Additional optional settings
modal secret create github-full-secret \
  GITHUB_TOKEN=your_pat_here \
  WEBHOOK_SECRET=your_webhook_secret_here \
  ALLOWED_REPOS="owner/repo1,owner/repo2" \
  RUNNER_VERSION="2.333.1" \
  RUNNER_GROUP_ID="1"

  5. Deploy the app:

    modal deploy app.py

  6. Configure the GitHub Webhook:

    • Go to your repository Settings > Webhooks > Add webhook.
    • Payload URL: Use the URL provided by modal deploy.
    • Content type: application/json.
    • Secret: Use the same WEBHOOK_SECRET you defined in step 2.
    • Events: Select Let me select individual events and check Workflow jobs.

  7. Update your GitHub Actions workflow:

    • Ensure the runs-on field includes modal and self-hosted.
    • Always include a unique label to prevent label contention between concurrent jobs.
    runs-on: [self-hosted, modal, "job-${{ github.run_id }}"]

    Without the unique label, multiple queued jobs compete for the same JIT runner and get stuck in "queued" state — each job needs its own dedicated runner.

    GPU jobs: Add a gpu: label to request GPU acceleration.

    runs-on: [self-hosted, modal, "job-${{ github.run_id }}", gpu:t4]

    Supported GPU types: gpu:t4, gpu:l4, gpu:a100, gpu:a100-80gb, gpu:h100

⚠️ Security Considerations

  • Trust Model: This runner executes with root privileges in isolated Modal sandboxes. Only allow trusted repositories via ALLOWED_REPOS.
  • JIT Tokens: Runner tokens are single-use and job-specific, limiting exposure if compromised.
  • Ephemeral Execution: Each job runs in a fresh sandbox that is destroyed after completion.
  • Webhook Verification: All requests are verified using HMAC-SHA256 signature validation.

⚠️ Limitations

  • Docker-in-Docker: Docker support is enabled via Modal's Alpha Docker-in-Sandbox feature (experimental_options={"enable_docker": True}). GitHub Actions services: and container actions should work, but this is an Alpha feature and may have edge cases.
  • Wiping State: Every job runs in a fresh sandbox. Files saved outside the repository workspace will be lost after the job completes.

How it Works

Every time a job is queued, Modal will spawn an ephemeral sandbox that runs the job and then exits. This ensures a clean and isolated environment for each job execution. The webhook is secured using HMAC-SHA256 signature verification.

Troubleshooting

Webhook signature verification fails (403)

WEBHOOK_SECRET is mismatched between the Modal secret and GitHub webhook settings. Make sure the same secret value is used in both modal secret create and the GitHub webhook configuration.

JIT token generation fails (401/403)

GITHUB_TOKEN lacks the required permissions or has expired. Use a fine-grained PAT with Actions: Read and Write and Administration: Read and Write permissions.

Sandbox spawn fails with timeout

The runner image build can take a while on first deploy, or the Docker-in-Sandbox setup may have failed. Check Modal logs for build errors. Ensure MODAL_IMAGE_BUILDER_VERSION=2025.06 is set (this is handled automatically in runner/config.py).

Jobs stuck in queued state

The webhook is not reaching the endpoint, or the modal label is missing from the workflow. Verify the webhook URL is correct and publicly accessible. Check that your workflow file has runs-on: [self-hosted, modal]. If using concurrent jobs, ensure each job has a unique label like job-${{ github.run_id }} to prevent label contention between JIT runners.

Multiple jobs stuck queued but one completes randomly

Label contention. Without unique labels in runs-on, a JIT runner created for job A can pick up job B instead. Fix: add "job-${{ github.run_id }}" to every job's runs-on.

GPU jobs fail to start

Either an invalid GPU label was used, or the GPU quota on your Modal account has been exceeded. Use one of the valid labels: gpu:t4, gpu:l4, gpu:a100, gpu:a100-80gb, gpu:h100. Check Modal GPU availability in your account.

Duplicate delivery warnings in logs

GitHub retries webhooks when responses are slow or network issues occur. This is normal. The deduplication cache handles it automatically, so no action is needed.

Environment Variables Reference

Variable Required Default Description
GITHUB_TOKEN Yes - GitHub PAT for runner registration
WEBHOOK_SECRET Yes - Secret for webhook signature validation
WEBHOOK_SECRET_OLD No - Previous secret for seamless rotation
ALLOWED_REPOS No (all) Comma-separated allowlist of owner/repo
RUNNER_VERSION No 2.334.0 GitHub Actions runner version
RUNNER_GROUP_ID No 1 Runner group ID
MAX_CONCURRENT_PER_REPO No (unlimited) Max concurrent sandboxes per repo
ALLOWED_CIDRS No (allow all) Comma-separated CIDR ranges for outbound
BLOCK_NETWORK No false Fully isolate sandbox network
CACHE_VOLUME_NAME No - Modal Volume name for persistent /cache mount
MODAL_REGION No - Modal region for sandbox deployment
SANDBOX_EXTRA_ENV No - JSON string of extra env vars for sandboxes
GITHUB_ENTERPRISE_DOMAIN No - Custom domain for GitHub Enterprise