Cron delivery silent failure + Telegram polling conflict + OpenRouter cascade failure

[minhaajre]

Description

Multiple issues observed with Hermes Gateway causing cron job outputs to silently fail delivery to Telegram, combined with OpenRouter credential exhaustion breaking the primary model and all auxiliary services simultaneously.

Issues

1. Silent Telegram delivery failure (live adapter bug)

Cron jobs complete successfully and log "delivered to telegram:<chat_id> via live adapter" — but the message never arrives. The live adapter object exists in memory but the underlying Telegram socket is disconnected (due to DNS failure or polling conflict). The adapter's send() does not raise an exception on a dead connection, so adapter_ok stays True, delivery is logged as successful, and the fallback standalone HTTP path is never reached.

Relevant code path: cron/scheduler.py lines ~588–629

2. Telegram polling conflict — multiple gateway instances

When the gateway crashes and launchd/systemd restarts it, the old Telegram polling connection is not fully torn down before the new one starts. This causes repeated:

Conflict: terminated by other getUpdates request; make sure that only one bot instance is running

Both instances then fail to poll, and any cron delivery during this window is dropped.

3. OpenRouter credential pool — cascading failure

When all OpenRouter keys in the credential pool are exhausted or invalid (401/429), the entire system fails: primary model, all 10 auxiliary services (compression, vision, session_search, etc.) are all routed through OpenRouter. A single provider failure takes everything down simultaneously with no fallback to other configured providers (Nous, Copilot, OpenAI Codex).

Steps to Reproduce

Issue 1:

1. Start gateway with Telegram connected
2. Trigger a network blip or DNS failure that drops the Telegram socket
3. Wait for a cron job to fire
4. Observe: agent log says delivered to telegram via live adapter, but message never arrives on Telegram

Issue 2:

1. Configure gateway as a launchd service with KeepAlive: { SuccessfulExit: false }
2. Cause the gateway to crash (e.g. OpenRouter 429 flood)
3. Observe repeated polling conflict warnings in gateway.log as old and new instances race

Issue 3:

1. Configure all auxiliary providers to use openrouter
2. Have both OpenRouter keys in the credential pool go exhausted simultaneously
3. Observe: primary model fails, all auxiliary services fail, no fallback occurs

Expected Behavior

1. If the live adapter send fails silently (no exception but connection is dead), fall through to the standalone HTTP delivery path
2. Gateway restart should cleanly terminate the previous Telegram polling session before starting a new one
3. When OpenRouter is unavailable, auxiliary services should fall back to other configured providers (Nous, Copilot, etc.)

Actual Behavior

1. Silent delivery failure logged as success
2. Polling conflict storm, both instances eventually time out
3. Complete system failure when OpenRouter credential pool is exhausted

Logs / Error Output

INFO cron.scheduler: Job 'AI Tools & Features Daily Brief' completed successfully
INFO cron.scheduler: Job 'dcad58f7f082': delivered to telegram:6456820456 via live adapter
# ^ message never arrived on Telegram

WARNING gateway.platforms.telegram: Telegram polling conflict (1/3), will retry in 10s. Error: Conflict: terminated by other getUpdates request
WARNING agent.auxiliary_client: Auxiliary: marking openrouter unhealthy for 60s (payment / credit error)
WARNING agent.auxiliary_client: resolve_provider_client: openrouter requested but OpenRouter credential pool has no usable entries
WARNING root: Fallback to openrouter failed: provider not configured

Environment

• OS: macOS 15 (Darwin 25.5.0)
• Hermes: latest (gateway mode, launchd service)
• Telegram: polling mode (not webhook)
• OpenRouter: free-tier models (arcee-ai/trinity-large-thinking:free, openai/gpt-oss-20b:free)
• Multiple providers configured in credential pool: openrouter, nous, copilot, openai-codex

[minhaajre]

Hi @impetus82 — thanks for the quick response and the scoped breakdown.

The focused slice makes sense. The dead-adapter false-positive is the most painful one since it silently eats messages with no user-visible signal.

A few clarifications before you start:

1. Test command — there's no existing test suite for the delivery path. A small standalone harness (e.g. a mock adapter that simulates a disconnected socket, verifies fallback to the HTTP path, and checks last_delivery_error is populated) would be the right form. Whatever you think is most practical to run in CI.

2. Scope confirm — agreed on leaving polling-conflict and OpenRouter cascade as follow-ups. Just the live adapter health check + fallback + visible failure for this slice.

3. Gateway version — latest from this repo's main branch. The relevant code is in cron/scheduler.py around lines 584–634.

Go ahead when ready. Happy to answer questions as you work through it.

[minhaajre]

Sorry i was under impression that a team handles these issues on voluntarily basis. I am not willing to pay for this. Thanks for taking the time to review this though.

[outsourc-e]

This issue describes Hermes Agent gateway behavior (cron delivery, Telegram polling, OpenRouter cascade), not Hermes Workspace UI/server behavior. Closing as misfiled — please re-open in the hermes-agent repo if this is still relevant.