Automation

.dockerignore

@@ -0,0 +1,21 @@
+# Foundry build artifacts
+out/
+cache/
+dependencies/
+
+# Git
+.git/
+
+# Node modules (rebuilt in Docker)
+node_modules/
+
+# Test and deploy scripts (not needed in actions container)
+test/
+script/
+src/contracts/
+
+# Misc
+.env
+.env.*
+coverage/
+lcov.info

.env.example

@@ -8,6 +8,7 @@ MAINNET_URL=
 # [Optional] RPC URLs for other networks
 TESTNET_URL=
 SONIC_URL=
+HOLESKY_URL=
 LOCAL_URL=
 
 # ╔══════════════════════════════════════════════════════════════════════════════╗
@@ -39,15 +40,22 @@ FORK_BLOCK_NUMBER_SONIC=
 # DEPLOYER_ADDRESS=
 
 # ╔══════════════════════════════════════════════════════════════════════════════╗
-# ║                          OPENZEPPELIN DEFENDER                               ║
+# ║                              AUTOMATON RUNTIME                               ║
 # ╚══════════════════════════════════════════════════════════════════════════════╝
 
-# Relayer credentials (for automated transactions)
-# DEFENDER_RELAYER_KEY=
-# DEFENDER_RELAYER_SECRET=
+# Bearer token the parent admin API uses to authenticate to the runner
+# ACTION_API_BEARER_TOKEN=
 
-# Team API credentials (for managing Defender Actions)
-# Used with: npx hardhat setActionVars, npx defender-autotask update-code
-# Note: Different from Relayer credentials
-# DEFENDER_TEAM_KEY=
-# DEFENDER_TEAM_SECRET=
+# Base URL the runner advertises back to automaton for manual triggers
+# RUNNER_BASE_URL=http://arm-oeth:8080
+
+# Shared Postgres URL (nonce queue + schedules + runs)
+# DATABASE_URL=
+
+# ╔══════════════════════════════════════════════════════════════════════════════╗
+# ║                                AWS KMS SIGNER                                ║
+# ╚══════════════════════════════════════════════════════════════════════════════╝
+# needed for running scheduled actions via hardhat
+
+# AWS_ACCESS_KEY_ID=
+# AWS_SECRET_ACCESS_KEY=

.github/workflows/actions-image.yml

@@ -0,0 +1,60 @@
+name: Actions Image
+
+on:
+  push:
+    branches:
+      - automation
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  packages: write
+
+jobs:
+  build-and-push:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set up SSH agent for @automaton/client deploy key
+        uses: webfactory/ssh-agent@v0.9.0
+        with:
+          ssh-private-key: ${{ secrets.TALOS_DEPLOY_KEY }}
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Log in to GitHub Container Registry
+        uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Prepare image metadata
+        id: prep
+        run: |
+          IMAGE_NAME="ghcr.io/${{ github.repository_owner }}/arm-actions"
+          IMAGE_NAME="$(echo "${IMAGE_NAME}" | tr '[:upper:]' '[:lower:]')"
+          echo "image_name=${IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
+
+      - name: Build and push image
+        uses: docker/build-push-action@v5
+        with:
+          context: .
+          file: ./dockerfile-actions
+          push: true
+          ssh: default
+          tags: |
+            ${{ steps.prep.outputs.image_name }}:latest
+            ${{ steps.prep.outputs.image_name }}:${{ github.sha }}
+
+      - name: Trigger Railway redeploy
+        if: success() && env.RAILWAY_TOKEN != ''
+        env:
+          RAILWAY_TOKEN: ${{ secrets.RAILWAY_TOKEN }}
+          RAILWAY_SERVICE_ID: ${{ secrets.RAILWAY_SERVICE_ID }}
+        run: |
+          npm i -g @railway/cli
+          railway redeploy --service "$RAILWAY_SERVICE_ID" --yes

README.md

@@ -293,6 +293,29 @@ yarn hardhat tenderlySync --network sonic
 yarn hardhat tenderlyUpload --network sonic --name ORIGIN_ARM
 ```
 
+## Automated Actions (Talos) 
+
+The `src/js/tasks/actions/*.ts` files are hardhat tasks that handle operational jobs (allocations, fee collection, withdrawal requests, etc.). In production they're driven by a container that imports [`@talos/client`](https://github.com/oplabs/talos):
+
+- **`runner.ts`** at repo root calls `runContainer({ product: "arm-oeth", workdir: "/app" })`. The library reads enabled schedules from the shared Talos Postgres, fires them via croner, and spawns each schedule's command as `pnpm hardhat <name> --network <chain>`.
+- **`migrations/seed_schedules.sql`** is a one-time seed of the `schedules` table mirroring the old `cron/cron-jobs.ts`.
+- **`src/js/tasks/lib/action.ts`** wraps the hardhat signer with `wrapSignerWithNonceQueueV6` from the library when `DATABASE_URL` is set. That routes `signer.sendTransaction` through Postgres row-locked nonce coordination.
+
+### Running actions locally
+
+Every action remains directly executable as a hardhat task on your dev machine:
+
+```bash
+pnpm hardhat allocateLido --network mainnet
+pnpm hardhat healthcheck
+```
+
+**No Postgres required.** The library's nonce queue is gated by `process.env.DATABASE_URL`: if unset, the action uses a raw ethers signer with the provider's default nonce handling — exactly the behavior you had before Automaton existed. The gate is a single check at the top of the handler; no DB connection is opened.
+
+If you opt in by setting `DATABASE_URL` (e.g., via `docker compose up`), the nonce queue engages and will try to connect to whatever that URL points at. `unset DATABASE_URL` to go back to the unwrapped path.
+
+Signer selection (`DEPLOYER_PRIVATE_KEY` → KMS via `KMS_RELAYER_ID` → `IMPERSONATE` → Defender → hardhat first signer) lives entirely in `src/js/utils/signers.ts` and is unchanged.
+
 ## Open Zeppelin Defender
 
 [Open Zeppelin Defender v2](https://docs.openzeppelin.com/defender/v2/) is used to manage the Operations account and automate AMM operational jobs like managing liquidity.
@@ -308,7 +331,7 @@ For this to work, a Defender Team API key with `Manage Actions` capabilities is
 
 The following will set the Defender Team API key and bundle the Actions code ready for upload.
 
-```bash
+```bash 
 # Set the DEFENDER_TEAM_KEY and DEFENDER_TEAM_SECRET env vars in the .env file
 
 yarn rollup -c src/js/actions/rollup.config.cjs

docker-compose.yml

@@ -0,0 +1,36 @@
+services:
+  postgres:
+    image: postgres:16-alpine
+    environment:
+      POSTGRES_DB: nonce_queue
+      POSTGRES_USER: nonce
+      POSTGRES_PASSWORD: nonce
+    volumes:
+      - pgdata:/var/lib/postgresql/data
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U nonce -d nonce_queue"]
+      interval: 5s
+      timeout: 3s
+      retries: 5
+
+  actions:
+    build:
+      context: .
+      dockerfile: dockerfile-actions
+    depends_on:
+      postgres:
+        condition: service_healthy
+    ports:
+      - "8080:8080"
+    environment:
+      - DATABASE_URL=${DATABASE_URL:-postgresql://nonce:nonce@postgres:5432/nonce_queue}
+      - MAINNET_URL=${MAINNET_URL:-}
+      - SONIC_URL=${SONIC_URL:-}
+      - HOLESKY_URL=${HOLESKY_URL:-}
+      - ACTION_API_BEARER_TOKEN=${ACTION_API_BEARER_TOKEN:-test-token}
+      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID:-}
+      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY:-}
+      - RUNNER_BASE_URL=${RUNNER_BASE_URL:-http://arm-oeth:8080}
+
+volumes:
+  pgdata:

dockerfile-actions

@@ -0,0 +1,47 @@
+# syntax=docker/dockerfile:1.7
+FROM oven/bun:1
+
+ENV DEBIAN_FRONTEND=noninteractive \
+  DEBUG_HIDE_DATE=true \
+  NODE_OPTIONS=--max-old-space-size=8096 \
+  PNPM_STORE_DIR=/root/.local/share/pnpm/store
+
+RUN apt-get update \
+  && apt-get install -y --no-install-recommends \
+  ca-certificates \
+  curl \
+  git \
+  openssh-client \
+  build-essential \
+  python3 \
+  bsdutils \
+  nodejs \
+  npm \
+  && rm -rf /var/lib/apt/lists/*
+
+RUN mkdir -p /root/.ssh \
+  && ssh-keyscan -t ed25519,rsa github.com >> /root/.ssh/known_hosts
+
+# Install Foundry (forge/cast/anvil) — required at hardhat config load time
+# because hardhat.config.js loads @nomicfoundation/hardhat-foundry.
+ENV PATH="/root/.foundry/bin:${PATH}"
+RUN curl -L https://foundry.paradigm.xyz | bash \
+  && /root/.foundry/bin/foundryup
+
+WORKDIR /app
+
+# Install dependencies first for better caching. Requires BuildKit SSH
+# forwarding (e.g. Railway deploy key) to resolve @automaton/client.
+COPY package.json pnpm-lock.yaml ./
+RUN --mount=type=ssh \
+    --mount=type=cache,id=arm-oeth-pnpm-store,target=/root/.local/share/pnpm/store \
+    npm install -g pnpm && pnpm install --frozen-lockfile
+
+# Copy the rest of the workspace.
+COPY . .
+
+ENV MAINNET_URL="" \
+  SONIC_URL="" \
+  HOLESKY_URL=""
+
+CMD ["bun", "run", "runner.ts"]

docs/automaton-observability.md

@@ -0,0 +1,126 @@
+# Automaton observability cookbook
+
+How to answer operational questions about the [Automaton](../cron/README.md) using Loki + Grafana. The same schema is emitted by the sibling Automaton in `origin-dollar` — swap `app="arm-oeth"` for `app="origin-dollar"` and every query below works there too.
+
+## Event model
+
+Every scheduled invocation produces:
+
+1. **Exactly one `action.start`** from the supervisor, the moment the child is spawned.
+2. **Exactly one terminal event** from the supervisor: `action.success` (exit 0) or `action.failure` (non-zero exit, signal, or spawn error).
+3. **Optionally, one `action.error`** from the task wrapper if it threw — carries the error class, message, stack, chain, and network.
+4. Any number of in-flight `info`/`debug` lines the task itself emitted.
+
+All events for one run share the same `run_id`. Use it as the join key when investigating.
+
+The supervisor owns events 1 and 2. **Even if the child crashes, OOMs, segfaults, or never starts**, the supervisor still produces both — counts and rates stay correct on the worst days.
+
+## Field schema
+
+| Field | Type | Label? | Appears on | Notes |
+|---|---|---|---|---|
+| `app` | string | **label** | all | Always `arm-oeth` (or `origin-dollar` in the sibling repo). |
+| `action` | string | **label** | all | The job name from `cron-jobs.ts`. ~30 distinct values. |
+| `event` | string | **label** | start/success/failure/error | One of `action.start`, `action.success`, `action.failure`, `action.error`. |
+| `source` | string | **label** | start/success/failure/error | `supervisor` for run-lifecycle events, `task` for `action.error`. |
+| `run_id` | UUID | field | all | Correlation key. Field, not label, to keep cardinality bounded. |
+| `duration_ms` | number | field | success/failure/error | Wall-clock from supervisor spawn (or task wrapper start) to terminal event. |
+| `exit_code` | number\|null | field | success/failure | From the child process. `null` on spawn failure. |
+| `signal` | string\|null | field | success/failure | e.g. `SIGKILL` (often = OOM-killed when paired with code 137). |
+| `spawn_failed` | bool | field | failure | Set when the child couldn't be spawned at all. |
+| `schedule` | string | field | start | The cron expression for the job. |
+| `command` | string | field | start | The shell command being executed. |
+| `chain_id` | number | field | error | Resolved chain id, if signer setup got that far. |
+| `network` | string | field | error | Human network name (`mainnet`, `sonic`, …). |
+| `error_name` | string | field | error | Error class (`Error`, `ProviderError`, …). |
+| `error_message` | string | field | error | The thrown message. |
+| `error_stack` | string | field | error | Full JS stack trace. |
+
+**Why labels are restricted to `action`, `event`, `source`:** Loki indexes labels, so high-cardinality labels blow up the index. `run_id` (UUIDs), `error_message` (free text), and numeric fields like `duration_ms` would all be cardinality bombs. They're still queryable via `| json` extraction; you just can't use them in stream selectors.
+
+## Common questions → queries
+
+### How many runs of X happened this week?
+
+```logql
+sum(count_over_time({app="arm-oeth", action="autoRequestLidoWithdraw", event="action.start"}[7d]))
+```
+
+### How many of those failed?
+
+```logql
+sum(count_over_time({app="arm-oeth", action="autoRequestLidoWithdraw", event="action.failure"}[7d]))
+```
+
+### Why did a particular run fail?
+
+Find recent failures, then pivot on `run_id`:
+
+```logql
+{app="arm-oeth", event="action.error"}
+  | json
+  | line_format "{{.action}} [{{.run_id}}] {{.error_name}}: {{.error_message}}"
+```
+
+Pick a `run_id` from the result, then:
+
+```logql
+{app="arm-oeth"} | json | run_id="<that-uuid>"
+```
+
+That returns the full breadcrumb in time order: `action.start` → in-flight info lines from the task → `action.error` (with stack) → supervisor's `action.failure` (with exit code/signal). Usually enough to fix without rerunning locally.
+
+### Average runtime per action
+
+```logql
+avg by (action) (
+  avg_over_time(
+    {app="arm-oeth", event=~"action.success|action.failure", source="supervisor"}
+      | json | unwrap duration_ms [7d]
+  )
+)
+```
+
+p95 instead:
+
+```logql
+quantile_over_time(0.95,
+  {app="arm-oeth", event=~"action.success|action.failure", source="supervisor"}
+    | json | unwrap duration_ms [7d]
+) by (action)
+```
+
+### Success rate per action
+
+```logql
+sum by (action) (count_over_time({app="arm-oeth", event="action.success"}[7d]))
+/
+sum by (action) (count_over_time({app="arm-oeth", event="action.start"}[7d]))
+```
+
+Using `action.start` as the denominator (rather than success+failure) catches the pathological case where a run started but no terminal event was ever emitted — the ratio drops below 1.0 and you notice.
+
+### Crashed-without-explanation runs
+
+Runs that produced an `action.failure` with **no** preceding `action.error`. These are the ones where the child died before reaching the task wrapper (OOM, parse error, missing env var, signal). Fewest moving parts:
+
+```logql
+{app="arm-oeth", event="action.failure", source="supervisor"} | json | exit_code=137
+```
+
+Code 137 + `SIGKILL` is almost always OOM. Code 1 with no `action.error` for the same `run_id` usually means the hardhat process failed before importing the task wrapper.
+
+## Known limitation: log loss on hard kill
+
+Log shipping is direct from the supervisor process to Loki via winston-loki's HTTP transport, batched on a 5ms interval. On graceful `SIGTERM` we `await flushLogger()` before exiting. On `SIGKILL` (OOM-killer, `kill -9`, container force-stop) the in-flight batch dies with the process and those events are lost. Fully fixing this would require a stdout-scraping sidecar (promtail / grafana-agent / alloy) — out of scope. In practice the gap is small: supervisor lifecycle events are emitted at job boundaries, not mid-job, so the window for loss is narrow.
+
+## When to graduate to Prometheus
+
+Symptoms that mean log queries have outgrown their usefulness:
+
+- Grafana panels are slow to render because they re-scan the log stream.
+- You want recording rules so dashboards don't recompute on every refresh.
+- You want alerts on "no successful run in the last N minutes" with cheap evaluation.
+- You want long-retention metrics (months) without paying log-storage costs.
+
+When that happens, add a `/metrics` endpoint to `cron-supervisor.ts` exposing `automaton_runs_total{action,status}`, `automaton_run_duration_seconds{action}` (histogram), and `automaton_last_success_timestamp_seconds{action}` (gauge). The log schema stays as-is for debugging — metrics complement it, they don't replace it.