Skip to content

fix: take lock in PriorityJobQueue.is_done - #101

Merged
bradjin8 merged 4 commits into
developfrom
fix/queue-is-done-lock
Jul 31, 2026
Merged

fix: take lock in PriorityJobQueue.is_done#101
bradjin8 merged 4 commits into
developfrom
fix/queue-is-done-lock

Conversation

@bradjin8

@bradjin8 bradjin8 commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Closes #100

Summary

  • PriorityJobQueue.is_done now acquires self._lock when reading _jobs and _completed_keys, matching is_empty and other accessors
  • Adds test_concurrent_is_done_invariant to stress concurrent enqueue/complete while a monitor thread checks that is_done is only True when every job is terminal
  • Changelog entry under [Unreleased] -> ### Fixed

Test plan

  • pytest tests/test_queue.py::TestThreadSafety -v
  • pytest -m "not integration" (full unit suite)
  • pre-commit run -a

Notes

completed_count, running_count, and total_jobs remain unlocked; left for a follow-up per issue scope.

Summary by CodeRabbit

  • Bug Fixes
    • Fixed queue completion checks during concurrent job activity.
    • Dependency-blocked jobs are now automatically re-queued when their dependencies finish or are cancelled.
    • Improved cleanup of cancelled jobs to prevent stale queue state.
  • Tests
    • Expanded concurrency testing for queue completion, dependency handling, cancellation, and worker termination.
  • Documentation
    • Updated the unreleased changelog with the queue reliability fixes.
  • Chores
    • Added a timeout to the continuous integration test job.

@bradjin8
bradjin8 requested a review from wpak-ai as a code owner July 30, 2026 20:27
@bradjin8 bradjin8 self-assigned this Jul 30, 2026
@coderabbitai

coderabbitai Bot commented Jul 30, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 49d3c260-4d6e-4d8f-87c1-f2e62935abe2

📥 Commits

Reviewing files that changed from the base of the PR and between e62ad81 and 4eb630b.

📒 Files selected for processing (3)
  • CHANGELOG.md
  • cli/localci/core/queue.py
  • cli/tests/test_queue.py
🚧 Files skipped from review as they are similar to previous changes (3)
  • cli/localci/core/queue.py
  • cli/tests/test_queue.py
  • CHANGELOG.md

📝 Walkthrough

Walkthrough

PriorityJobQueue now requeues dependency-ready jobs after job state changes and evaluates is_done under the queue lock. Tests cover dependency promotion, lock contention, concurrent completion, and bounded thread shutdown.

Changes

Queue state synchronization

Layer / File(s) Summary
Promote dependency-ready jobs
cli/localci/core/queue.py
Terminal state changes and cancel-all operations requeue current-priority jobs in WAITING_DEPS once all dependencies are completed. The new _promote_waiting_deps helper checks job dependencies and marks eligible jobs as QUEUED.
Synchronize completion-state reads
cli/localci/core/queue.py, CHANGELOG.md
PriorityJobQueue.is_done now evaluates its completion predicate while holding self._lock. The changelog records both queue fixes.
Validate queue state and concurrency
cli/tests/test_queue.py
Tests import QueuedJobStatus, define shared concurrency constants and thread helpers, and add coverage for dependency promotion, lock contention on is_done, cancellation state cleanup, concurrent producer/consumer operations, and bounded consumer shutdown.
Bound test-job execution
.github/workflows/ci.yml
The CI test job has a 15-minute execution timeout.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Suggested reviewers: wpak-ai, clean6378-max-it

Sequence Diagram(s)

sequenceDiagram
  participant QueueOp as Queue Operation
  participant Queue as PriorityJobQueue
  participant Job as Job
  QueueOp->>Queue: complete, skip, cancel, or cancel_all
  Queue->>Queue: _promote_waiting_deps()
  Queue->>Job: check all dependencies completed
  alt Dependencies ready
    Queue->>Job: mark WAITING_DEPS as QUEUED
  end
Loading
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Out of Scope Changes check ⚠️ Warning The PR adds WAITING_DEPS promotion and _running_keys cleanup, which are queue behavior changes not required by linked issue #100. Move these fixes to separate linked issues or document them as explicit scope with acceptance criteria.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the primary change: locking the complete PriorityJobQueue.is_done read.
Linked Issues check ✅ Passed The PR locks the full is_done read and adds concurrent stress coverage; it does not change orchestrator dispatch code [#100].
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/queue-is-done-lock

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@cli/tests/test_queue.py`:
- Around line 367-375: Fix the TOCTOU race in the monitor function by
re-checking queue.is_done after obtaining the get_all_jobs() snapshot, and only
recording “is_done True with non-terminal jobs” when is_done remains true.
Preserve the existing empty-queue check and terminal-status validation, while
avoiding false violations caused by jobs enqueued between the two calls.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 303a4c71-68e0-4ab8-a538-c5f61b12d85f

📥 Commits

Reviewing files that changed from the base of the PR and between a9aa45a and 737c2b5.

📒 Files selected for processing (3)
  • CHANGELOG.md
  • cli/localci/core/queue.py
  • cli/tests/test_queue.py

Comment thread cli/tests/test_queue.py Outdated
@clean6378-max-it

Copy link
Copy Markdown
Collaborator

queue.py:233 - move WAITING_DEPS jobs back to QUEUED once their dependencies finish, the way line 339 already does for WAITING_PRIORITY (nothing ever moves a job out of WAITING_DEPS. Two jobs linked by needs land at the same priority by default, so the dependent is stranded, the level never completes, and _dispatch_loop spins forever) pre-existing

Comment thread cli/tests/test_queue.py Outdated
Comment thread cli/tests/test_queue.py Outdated
Comment thread cli/tests/test_queue.py Outdated
Comment thread cli/tests/test_queue.py
@bradjin8

Copy link
Copy Markdown
Collaborator Author

queue.py:233 - move WAITING_DEPS jobs back to QUEUED once their dependencies finish, the way line 339 already does for WAITING_PRIORITY (nothing ever moves a job out of WAITING_DEPS. Two jobs linked by needs land at the same priority by default, so the dependent is stranded, the level never completes, and _dispatch_loop spins forever) pre-existing

Added _promote_waiting_deps() (mirrors the WAITING_PRIORITY -> QUEUED pattern at line 339) and call it from mark_completed, mark_skipped, cancel, and cancel_all before _check_priority_advance.
Only jobs at the current priority level are promoted.

@clean6378-max-it

Copy link
Copy Markdown
Collaborator

Two things left.

  1. cancel_all doesn't discard from _running_keys. READY jobs are cancellable and next_ready has already put the key in _running_keys, but the loop only adds to _completed_keys. On this branch: enqueue 3, one next_ready(), then cancel_all() gives running_count == 1 and pending_count == -1. It shows up as a phantom running job in the summary after stop_on_first_failure. Same discard(key) that mark_completed already does. test_cancel_all never calls next_ready, so it needs that case with running_count == 0 and pending_count >= 0.

  2. holder.join(timeout=5) needs an assert not holder.is_alive() after it. Without it a stuck holder passes green and leaves a live thread that blocks pytest at exit. The consumer tests already do this.

Minor, not a blocker: checker.join(timeout=0.05) can't tell "blocked on the lock" from "not scheduled yet", so on a busy runner the guard passes without checking anything. An Event set just before the is_done read would close it.

Unrelated to your change: all_dependencies_met only checks _completed_keys, which fills on failure too, so a dependent runs after its dependency fails. It predates this PR. I'll open a separate issue.

@bradjin8
bradjin8 merged commit 09ebd73 into develop Jul 31, 2026
7 checks passed
@bradjin8
bradjin8 deleted the fix/queue-is-done-lock branch July 31, 2026 16:30
@bradjin8
bradjin8 restored the fix/queue-is-done-lock branch July 31, 2026 17:43
@bradjin8
bradjin8 deleted the fix/queue-is-done-lock branch July 31, 2026 18:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Take the lock in PriorityJobQueue.is_done

2 participants