Skip to content

[FLINK-40043] fix the paimon pipeline job couldn’t increase job parallelism.#4457

Open
sd4324530 wants to merge 1 commit into
apache:masterfrom
sd4324530:FLINK-40043
Open

[FLINK-40043] fix the paimon pipeline job couldn’t increase job parallelism.#4457
sd4324530 wants to merge 1 commit into
apache:masterfrom
sd4324530:FLINK-40043

Conversation

@sd4324530

Copy link
Copy Markdown
Contributor

Fixed the issue where Paimon pipeline job couldn’t increase job parallelism.

Reproduction method:

  1. Stop the CDC job using a savepoint
  2. Increase the parallelism of the job
  3. Start the job using a savepoint

Error message:

java.util.NoSuchElementException
	at java.base/java.util.ArrayList$Itr.next(ArrayList.java:1000)
	at org.apache.flink.cdc.connectors.paimon.sink.v2.PaimonSink.restoreWriter(PaimonSink.java:101)
	at org.apache.flink.streaming.runtime.operators.sink.StatefulSinkWriterStateHandler.createWriter(StatefulSinkWriterStateHandler.java:120)
	at org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator.initializeState(SinkWriterOperator.java:153)
	at org.apache.flink.cdc.runtime.operators.sink.DataSinkWriterOperator.initializeState(DataSinkWriterOperator.java:132)
	at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:147)
	at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:294)
	at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:106)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreStateAndGates(StreamTask.java:858)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$restoreInternal$5(StreamTask.java:812)
	at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:812)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:771)
	at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:970)
	at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:939)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:763)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
	at java.base/java.lang.Thread.run(Thread.java:829)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant