feat(evolution): process every 4h (fix backlog growth: ~25 gen/day vs ~1-2 proc/night)#461
Merged
Merged
Conversation
…acklog growth)
The pipeline generates ~25 issues/day (research+issues+introspection) but
the once-daily processing chain landed only ~1-2/night (analysis caps at 8,
most triage out / don't land cleanly), so the open backlog grows ~20/day —
the recurring 'again many unprocessed issues'.
Run the PROCESSING chain every 4h instead of daily (generation stays daily):
- analysis: 0 1,5,9,13,17,21 * * * (was 0 21)
- implementation: 0 2,6,10,14,18,22 * * * (was 0 22, +1h after analysis)
- integration: 0 3,7,11,15,19,23 * * * (was 0 23, +1h so CI settles)
This is ~6x processing passes/day. Mechanically safe: reports are
date-keyed (overwritten each run, latest wins) and the implementation
freshness gate is date-based ('today's analysis = fresh'), so intra-day
re-runs don't break the chain; the accepted-label + dedup prevent
re-processing already-handled issues.
Watchdog: STAGES now mirrors the FIRST daily slot (analysis=1, impl=2,
integration=3) — 'a report for today must exist by then' (reports are
date-keyed). Mirror test updated to read the first hour of a multi-slot
cron; stage-report tests made slot-aware. 35 watchdog tests pass.
NOTE: takes effect after the server re-registers cron (register_evolution_cron
on next 'hermes update'/pull). Cost ~6x the processing agent runs/day — dial
to every 6h/8h if needed. Complementary generation backlog-cap still
recommended to fully converge (generation ~25/day > even 6x processing).
Contributor
🔎 Lint report:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why (root cause of "again many unprocessed issues")
Measured generation vs processing:
(*much of 06-20/21 "completed" was manual curation, not the autonomous cycle.) Last night the autonomous chain ran fine but landed exactly 1 PR (#436). The pipeline GENERATES ~25 issues/day but the once-daily processing lands ~1-2 → the board grows ~20/day. Not a breakage — a funnel imbalance that recurs every day.
What
Run the processing chain every 4h (generation stays daily):
analysis:0 1,5,9,13,17,21(was0 21)implementation:0 2,6,10,14,18,22(was0 22, +1h after analysis)integration:0 3,7,11,15,19,23(was0 23, +1h so CI settles)~6× processing passes/day. Mechanically safe (verified): reports are date-keyed (overwritten each run, latest wins), the implementation freshness gate is date-based (today's analysis = fresh), and
accepted-label + dedup stop re-processing handled issues. So intra-day re-runs just chew through the remaining backlog.Watchdog (kept consistent)
STAGESmirrors the first daily slot (analysis=1, impl=2, integration=3) — "a report for today must exist by then." Mirror test now reads the first hour of a multi-slot cron; stage-report tests made slot-aware. 35 watchdog tests pass.Caveats (honest)
register_evolution_cron, runs onhermes update/pull).Local verification
cron exprs valid 5-field; watchdog suite 35 passed.
parse_schedule/compute_next_runcan't run locally (croniter not installed in this env — affects the existing daily exprs too), so the cron-execution path is verified by CI + on the server.