fix(jindo): make async reads concurrency-safe#426
Closed
zjw1111 wants to merge 4 commits into
Closed
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR hardens the Jindo filesystem InputStream async read path against concurrency hazards by removing shared per-stream read state, ensuring callbacks complete exactly once (including on immediate perform() failure), and wiring opt-in remote OSS-backed Jindo tests into existing test binaries.
Changes:
- Reworked
JindoInputStream::ReadAsyncto use per-request shared state and to surface/complete onperform()failures. - Removed reused
std::string_viewmember fromJindoInputStream; made sync reads use local result state and clarifiedInputStream::ReadAsynclifetime contract. - Added an opt-in
PAIMON_ENABLE_JINDO_TESTSswitch and enabled Jindo variants in existing fs + integration test targets, including new OSS pagination + concurrent read coverage.
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
src/paimon/fs/jindo/jindo_file_system.h |
Removes shared std::string_view member used across reads. |
src/paimon/fs/jindo/jindo_file_system.cpp |
Introduces per-async-call completion/result state and handles perform() failures. |
include/paimon/fs/file_system.h |
Documents async read lifetime requirements for callers. |
src/paimon/fs/jindo/jindo_file_system_test.cpp |
Adds OSS-backed tests for listing pagination and concurrent sync/async reads. |
src/paimon/common/fs/file_system_test.cpp |
Enables Jindo test parameter under PAIMON_ENABLE_JINDO_TESTS. |
test/inte/write_and_read_inte_test.cpp |
Enables Jindo integration parameter under PAIMON_ENABLE_JINDO_TESTS. |
src/paimon/CMakeLists.txt |
Conditionally compiles/links Jindo fs tests and defines PAIMON_ENABLE_JINDO_TESTS. |
test/inte/CMakeLists.txt |
Defines PAIMON_ENABLE_JINDO_TESTS for the integration test target when enabled. |
CMakeLists.txt |
Adds PAIMON_ENABLE_JINDO_TESTS option and enforces prerequisite options. |
cmake_modules/DefineOptions.cmake |
Exposes PAIMON_ENABLE_JINDO_TESTS as a configurable option. |
CONTRIBUTING.md |
Adds Conventional Commits guidance and notes on avoiding force-push during review iteration. |
AGENTS.md |
Adds repository-specific agent guidance (new file). |
.github/PULL_REQUEST_TEMPLATE.md |
Updates PR title guidance to Conventional Commits format. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+19
to
21
| #include <atomic> | ||
| #include <cassert> | ||
| #include <utility> |
Collaborator
Author
|
Closing this PR because its head branch contained unrelated documentation commits. A clean replacement PR will follow from codex/fix-jindo-async-read. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: N/A
JindoInputStreamreused one memberstd::string_viewacross synchronous and concurrent asynchronous reads, and ignored failures returned byperform(). This could race between reads or leave a caller's completion callback unresolved.This change:
perform()failsPAIMON_ENABLE_JINDO_TESTSswitch and enables Jindo cases in the existing filesystem and write/read integration test binariesTests
cmake --build build --target paimon-fs-test -j64cmake --build build --target paimon-write-and-read-inte-test -j64FsType/FileSystemTest.TestFactoryCreator/1FileFormatAndFileSystem/WriteAndReadInteTest.TestAppendSimple/1pre-commit run --files ...git diff --checkAPI and Format
Clarifies the existing
InputStream::ReadAsynclifetime contract. No storage format or protocol changes.Documentation
No user-facing documentation changes.
Generative AI tooling
Generated-by: OpenAI Codex (GPT-5)