Flaky ReconfigurationIntegrationSpec: small CSV source can finish before pause

[Ma77Ball]

Task Summary

ReconfigurationIntegrationSpec's two CSV-sourced tests use a 100-row source (smallCsvScanOpDesc). The run can reach COMPLETED before the pauseWorkflow round-trip takes effect; when it does, PauseHandler.pauseWorkflow finds no running regions and emits the current state (COMPLETED) rather than PAUSED, so TestUtils.shouldReconfigure's Await.result(pausedReached, 10.seconds) times out and the test fails. The existing Retries mixin does not help because the race recurs on retry. Fix by using mediumCsvScanOpDesc (100k rows) so the run stays in flight when pause lands, matching the non-flaky sibling specs ReconfigurationSpec and PauseSpec. Surfaced on the amber-integration CI job (e.g. run 28017649934).

Task Type

• Test

[Neilk1021]

/take

[Neilk1021]

/untake

[Yicong-Huang]

Another occurrence on amber-integration (macos-latest, 17) in #6123 (a pure-rename PR, so unrelated to the diff): https://github.com/apache/texera/actions/runs/28724290967/job/85178802125?pr=6123

Same failure mode: the Python worker throws InvalidTransitionException: Cannot transit from PAUSED to COMPLETED while ReconfigurationIntegrationSpec pauses a workflow whose small source finishes first. The identical commit passed on amber-integration (ubuntu-latest, 17).

[Yicong-Huang]

Two more occurrences on amber-integration (macos-latest, 17) in #6123 (runs 28724290967 and 28724904090; the first passed on rerun). Ubuntu passes every time on the same commits.

Note the failure mode has shifted since this issue was filed: the spec no longer uses smallCsvScanOpDesc — it already has the warmup pass and a slowSource() (30 rows, 0.25s apart) — yet it still flakes. The race now surfaces inside the Python worker state machine:

core.architecture.managers.state_manager.InvalidTransitionException: Cannot transit from PAUSED to COMPLETED

i.e. the worker has acknowledged PAUSED and then the end-of-input path tries to drive it straight to COMPLETED. So the proposed bigger-input fix is likely insufficient; the pause/complete interaction in the Python worker (or the test's pause timing) needs to tolerate this ordering.