Skip to content

Fix duplicate scheduling in procedure execution#17902

Merged
jt2594838 merged 6 commits into
masterfrom
fix-procedure-duplicate-scheduling
Jun 16, 2026
Merged

Fix duplicate scheduling in procedure execution#17902
jt2594838 merged 6 commits into
masterfrom
fix-procedure-duplicate-scheduling

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Prevent duplicate procedure execution

  • Add a per-procedure execution guard so a duplicated scheduler entry cannot run the same Procedure instance concurrently.
  • Release the guard, active executor count, and worker state in the worker finally path.
  • Release locks when a scheduled procedure is no longer RUNNABLE.

Avoid duplicate lock wakeups

  • De-duplicate LockQueue waiters by procedure id before adding a waiting procedure.
  • Keep PipeTaskCoordinatorLock waiting until a permit is actually acquired, even if the thread is already interrupted.

Tests

  • Add coverage for duplicate procedure scheduling, duplicate LockQueue wakeups, interrupted PipeTaskCoordinatorLock acquisition, and completed PipeHandleLeaderChangeProcedure equality.

This PR has:

  • been self-reviewed.
    • concurrent read
    • concurrent write
    • concurrent read and write
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage.

Key changed/added classes (or packages if there are too many classes) in this PR
  • org.apache.iotdb.confignode.procedure.Procedure
  • org.apache.iotdb.confignode.procedure.ProcedureExecutor
  • org.apache.iotdb.confignode.procedure.scheduler.LockQueue
  • org.apache.iotdb.confignode.manager.pipe.coordinator.task.PipeTaskCoordinatorLock
Local verification
  • PASS: mvn -pl iotdb-core/confignode spotless:apply
  • FAIL: mvn -pl iotdb-core/confignode -Dtest=TestLockRegime,TestProcedureExecutor,PipeHandleLeaderChangeProcedureTest,PipeTaskCoordinatorLockTest test failed during compile before test execution because local generated/snapshot dependencies were missing classes such as TGetCommitProgressReq, TPullCommitProgressReq, CommitProgressKeeper, and RegionProgress.

@codecov

codecov Bot commented Jun 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 70.50360% with 41 lines in your changes missing coverage. Please review.
✅ Project coverage is 41.17%. Comparing base (8fad784) to head (bca866a).
⚠️ Report is 7 commits behind head on master.

Files with missing lines Patch % Lines
.../iotdb/confignode/procedure/ProcedureExecutor.java 68.51% 17 Missing ⚠️
...ode/procedure/impl/node/AbstractNodeProcedure.java 0.00% 6 Missing ⚠️
...dure/impl/pipe/AbstractOperatePipeProcedureV2.java 0.00% 6 Missing ⚠️
...db/confignode/procedure/TimeoutExecutorThread.java 75.00% 4 Missing ⚠️
...otdb/confignode/procedure/scheduler/LockQueue.java 66.66% 2 Missing ⚠️
...otdb/db/schemaengine/table/DataNodeTableCache.java 50.00% 2 Missing ⚠️
...g/apache/iotdb/confignode/procedure/Procedure.java 80.00% 1 Missing ⚠️
.../impl/schema/AlterTimeSeriesDataTypeProcedure.java 0.00% 1 Missing ⚠️
...ntime/SubscriptionHandleLeaderChangeProcedure.java 0.00% 1 Missing ⚠️
.../procedure/scheduler/SimpleProcedureScheduler.java 94.44% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #17902      +/-   ##
============================================
+ Coverage     41.15%   41.17%   +0.02%     
  Complexity      318      318              
============================================
  Files          5258     5258              
  Lines        365613   365743     +130     
  Branches      47282    47300      +18     
============================================
+ Hits         150460   150596     +136     
+ Misses       215153   215147       -6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment on lines +110 to +115
boolean duplicated = blockingProcedure.awaitExecution(3, TimeUnit.SECONDS);

blockingProcedure.releaseExecutions(duplicated ? 2 : 1);
ProcedureTestUtil.waitForProcedure(procExecutor, procId);

Assert.assertFalse(duplicated);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

line 112 is weird together with line 115.

@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
B Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@jt2594838 jt2594838 merged commit 0a45a3b into master Jun 16, 2026
42 of 43 checks passed
@jt2594838 jt2594838 deleted the fix-procedure-duplicate-scheduling branch June 16, 2026 10:19
jt2594838 pushed a commit that referenced this pull request Jun 18, 2026
* Fix duplicate scheduling in procedure execution

* Fix delayed procedure deduplication and semaphore release

* Fix SQL parser error handler traversal

* Fix pipe procedure lock release race

* Fix procedure lock wait scheduling

(cherry picked from commit c25849a)
(cherry picked from commit 0a45a3b)
MileaRobertStefan pushed a commit to MileaRobertStefan/iotdb that referenced this pull request Jun 26, 2026
* Fix duplicate scheduling in procedure execution

* Fix delayed procedure deduplication and semaphore release

* Fix SQL parser error handler traversal

* Fix pipe procedure lock release race

* Fix procedure lock wait scheduling

(cherry picked from commit c25849a)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants