Skip to content

cron.md

Latest commit

 

History

History
176 lines (133 loc) · 5.74 KB

cron.md

File metadata and controls

176 lines (133 loc) · 5.74 KB

Cron and systemd-timer Deployment

recotem train is a plain process with a well-defined exit code contract. Any scheduler that can run a command on a schedule works.

Linux cron

Add to /etc/cron.d/recotem (or the user crontab via crontab -e). Source a secrets file so plaintext keys never appear in the crontab itself:

0 3 * * * recotem . /etc/recotem/secrets && /usr/local/bin/recotem train /etc/recotem/recipes/my_recipe.yaml >> /var/log/recotem/train.log 2>&1
# /etc/recotem/secrets — mode 600, owned by the cron user
export RECOTEM_SIGNING_KEYS="prod-2026-q2:aabbcc..."

Secure both the crontab and the secrets file:

chmod 600 /etc/cron.d/recotem
chown root:root /etc/cron.d/recotem
chmod 600 /etc/recotem/secrets
chown recotem:recotem /etc/recotem/secrets

DO NOT embed secrets directly in the crontab as inline env var assignments — /etc/cron.d/ files may be world-readable on some distributions:

# BAD — exposes the key to any local user who can read /etc/cron.d/recotem
RECOTEM_SIGNING_KEYS=prod-2026-q2:aabbcc...
0 3 * * * recotem /usr/local/bin/recotem train /etc/recotem/recipes/my_recipe.yaml >> /var/log/recotem/train.log 2>&1

Wrapper script

For more control over retries, alerting, and log rotation, use a wrapper script:

#!/usr/bin/env bash
# /usr/local/bin/recotem-train-daily.sh
set -euo pipefail

. /etc/recotem/secrets

RECIPE=/etc/recotem/recipes/my_recipe.yaml
LOG=/var/log/recotem/train-$(date +%Y%m%d-%H%M%S).log

/usr/local/bin/recotem train "$RECIPE" 2>&1 | tee "$LOG"
EXIT=${PIPESTATUS[0]}

case $EXIT in
  0) echo "train: success" ;;
  2) echo "train: RecipeError (check recipe YAML)" >&2; exit $EXIT ;;
  3) echo "train: DataSourceError (transient?)" >&2; exit $EXIT ;;
  4) echo "train: TrainingError (data or tuning issue)" >&2; exit $EXIT ;;
  5) echo "train: ArtifactError (check RECOTEM_SIGNING_KEYS)" >&2; exit $EXIT ;;
  6) echo "train: lock contested — another process holds the lock; retry later" >&2; exit $EXIT ;;
  7) echo "train: HTTP fetch error — network issue or sha256 mismatch; alert ops" >&2; exit $EXIT ;;
  8) echo "train: config error — check RECOTEM_SIGNING_KEYS and env vars; alert ops, do not retry" >&2; exit $EXIT ;;
  *) echo "train: unexpected error (exit $EXIT)" >&2; exit $EXIT ;;
esac
0 3 * * * recotem /usr/local/bin/recotem-train-daily.sh

systemd timer

A systemd timer gives better logging (journald), dependency handling, and restart control.

# /etc/systemd/system/recotem-train.service
[Unit]
Description=Recotem daily training
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
User=recotem
EnvironmentFile=/etc/recotem/secrets
ExecStart=/usr/local/bin/recotem train /etc/recotem/recipes/my_recipe.yaml
StandardOutput=journal
StandardError=journal
SyslogIdentifier=recotem-train
# /etc/systemd/system/recotem-train.timer
[Unit]
Description=Recotem daily training timer

[Timer]
OnCalendar=*-*-* 03:00:00 UTC
Persistent=true          # run on next boot if the last run was missed

[Install]
WantedBy=timers.target

Enable and start:

systemctl daemon-reload
systemctl enable --now recotem-train.timer

Check status:

systemctl status recotem-train.timer
journalctl -u recotem-train.service -n 50

Environment file

EnvironmentFile (systemd) or the secrets-sourcing pattern (cron) should be mode 600, owned by the service user, and excluded from version control.

# /etc/recotem/secrets — mode 600, owner recotem
RECOTEM_SIGNING_KEYS=prod-2026-q2:aabbcc...

serve as a systemd service

# /etc/systemd/system/recotem-serve.service
[Unit]
Description=Recotem serve
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=recotem
EnvironmentFile=/etc/recotem/secrets
Environment=RECOTEM_HOST=0.0.0.0
Environment=RECOTEM_PORT=8080
Environment=RECOTEM_LOG_FORMAT=json
Environment=RECOTEM_WATCH_INTERVAL=30
ExecStart=/usr/local/bin/recotem serve --recipes /etc/recotem/recipes/
Restart=on-failure
RestartSec=10
StandardOutput=journal
StandardError=journal
SyslogIdentifier=recotem-serve

[Install]
WantedBy=multi-user.target
systemctl enable --now recotem-serve.service

When recotem train writes a new artifact, the serve process detects it at the next poll and hot-swaps — no service restart needed.

Timezone

cron uses the system timezone (/etc/localtime); set CRON_TZ=UTC at the top of the crontab if you want explicit UTC. The systemd timer above pins OnCalendar=… UTC which is independent of system tz.

Lock contention

If a cron job fires while a previous training run is still active on the same host, the second invocation acquires no lock and exits 0 (skip). The lock uses POSIX flock, which is host-local — it does not coordinate runs across multiple machines, and with an s3:// / gs:// output.path it does not coordinate across pods either (see docs/deployment/k8s.md). Run recotem train from a single host (or guard cross-host concurrency in your scheduler) when you need single-writer semantics. Recotem logs recipe_lock_local_only whenever the output is a remote URI.

This is the default and is safe for standard cron setups but means the scheduler sees a successful run when nothing was actually trained — point alerting at the structured recipe_lock_contended_skipping log line, not just the exit code, or pass --fail-on-busy:

recotem train --fail-on-busy /etc/recotem/recipes/my_recipe.yaml

Exit will be non-zero when the lock is held, which most monitoring systems treat as a failure. Pair this with a cron schedule whose interval comfortably exceeds the p99 training duration; recotem train --quiet log lines include the run duration for sizing.