Fix write retry, pipe fallback, and snapshot recovery edge cases#18180
Open
Caideyipi wants to merge 3 commits into
Open
Fix write retry, pipe fallback, and snapshot recovery edge cases#18180Caideyipi wants to merge 3 commits into
Caideyipi wants to merge 3 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Keep tablet retries and failure reporting consistent
When an insert tablet spans multiple processors or time partitions, a type inconsistency in a later fragment used to replay already successful fragments. Track processed rows across the retry and rebuild only the remaining ranges.
Memory reservation can also fail after earlier tablet ranges or aligned/non-aligned row groups have reserved memory. Roll back those earlier reservations, preserve the original rejection status, and mark every range that was not inserted as failed so last-cache updates and consensus retry decisions do not treat missing rows as successful.
Make both V2 pipe builders lossless on fallback
PipeTableModelTsFileBuilderV2previously did not buffer data into its fallback builder, so a primary conversion failure could return an empty successful result. Both V2 builders now also delete primary-path files when switching to fallback, including files already sealed before a later database fails.The tree-model V2 builder now skips an individual null schema instead of dropping every valid column after it, and handles null DATE values without forcing fallback.
Replace all derived metadata during snapshot loading
Loading a template snapshot now replaces both template indexes, and an empty template-preset snapshot clears stale preset state.
QuotaInfo.clear()also clears the derived space-quota usage map so entries created after a snapshot do not survive recovery.Distinguish compatibility EOF from corrupted subscription snapshots
Commit-progress restoration uses full reads so legal short reads are accepted. A completely absent commit-progress section remains valid for older snapshots, while a partially present 1-3 byte size header now fails snapshot loading instead of silently discarding committed progress.
Verification
mvn spotless:apply -pl iotdb-core/node-commons,iotdb-core/datanode,iotdb-core/confignodeCommitProgressKeeperTest: 5 tests passedQuotaInfoTest,TemplateTableTest,TemplatePreSetTableTest: 6 tests passed with-DforkCount=0PipeTsFileBuilderV2Test: 4 tests passed with-DforkCount=0DataRegionTestmethods: 3 tests passed with-DforkCount=0TsFileProcessorTest: 16 tests passed with-DforkCount=0mvn -pl iotdb-core/node-commons,iotdb-core/datanode,iotdb-core/confignode -DskipTests -P with-zh-locale test-compileThis PR has:
Key changed/added classes (or packages if there are too many classes) in this PR
DataRegionandTsFileProcessorPipeTableModelTsFileBuilderV2andPipeTreeModelTsFileBuilderV2TemplateTable,TemplatePreSetTable, andQuotaInfoCommitProgressKeeper