Skip to content

Load: Optimized the downgraded logic for tsFile to insert more data when tsFile corrupted#17674

Merged
jt2594838 merged 6 commits into
masterfrom
load-query
Jun 23, 2026
Merged

Load: Optimized the downgraded logic for tsFile to insert more data when tsFile corrupted#17674
jt2594838 merged 6 commits into
masterfrom
load-query

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

As the title said.


This PR has:

  • been self-reviewed.
    • concurrent read
    • concurrent write
    • concurrent read and write
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods.
  • added or updated version, license, or notice information
  • added comments explaining the "why" and the intent of the code wherever would not be obvious
    for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold
    for code coverage.
  • added integration tests.
  • been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR

@luoluoyuyu luoluoyuyu left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary

Introduces LoadTreeTsFileTabletIterator and uses it in LoadTreeStatementDataTypeConvertExecutionVisitor so corrupted TsFiles can yield more insertable tablets on downgrade. Adds unit tests for the converter path.

Please fix failing datanode unit tests before merge.

Caideyipi added 3 commits June 3, 2026 16:15
# Conflicts:
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tsfile/parser/scan/TsFileInsertionEventScanParser.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/load/converter/LoadTreeStatementDataTypeConvertExecutionVisitor.java
@sonarqubecloud

sonarqubecloud Bot commented Jun 5, 2026

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@codecov

codecov Bot commented Jun 5, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 84.81013% with 48 lines in your changes missing coverage. Please review.
✅ Project coverage is 41.31%. Comparing base (a420681) to head (a3e00d5).
⚠️ Report is 104 commits behind head on master.

Files with missing lines Patch % Lines
...e/load/converter/LoadTreeTsFileTabletIterator.java 85.20% 37 Missing ⚠️
.../parser/query/TsFileInsertionEventQueryParser.java 76.31% 9 Missing ⚠️
...le/parser/scan/TsFileInsertionEventScanParser.java 92.30% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #17674      +/-   ##
============================================
+ Coverage     40.87%   41.31%   +0.43%     
+ Complexity     2611      318    -2293     
============================================
  Files          5186     5273      +87     
  Lines        351308   368255   +16947     
  Branches      44971    47655    +2684     
============================================
+ Hits         143585   152131    +8546     
- Misses       207723   216124    +8401     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@jt2594838
jt2594838 merged commit cb97fe4 into master Jun 23, 2026
42 of 43 checks passed
jt2594838 pushed a commit that referenced this pull request Jun 25, 2026
* Load: Optimized the downgraded logic for tsFile to insert more data when tsFile corrupted (#17674)

* down

* Update LoadTreeStatementDataTypeConvertExecutionVisitorTest.java

* Fix query parser deferred tablet failure handling

* Fix aligned load fallback corruption test

* Address load fallback review comments

(cherry picked from commit cb97fe4)

* Update LoadTreeStatementDataTypeConvertExecutionVisitorTest.java

* Update LoadTreeStatementDataTypeConvertExecutionVisitorTest.java
MileaRobertStefan pushed a commit to MileaRobertStefan/iotdb that referenced this pull request Jun 26, 2026
…hen tsFile corrupted (apache#17674)

* down

* Update LoadTreeStatementDataTypeConvertExecutionVisitorTest.java

* Fix query parser deferred tablet failure handling

* Fix aligned load fallback corruption test

* Address load fallback review comments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants