Skip to content

fix(jindo): make async reads concurrency-safe#426

Closed
zjw1111 wants to merge 4 commits into
alibaba:mainfrom
zjw1111:codex/reproduce-jindo-fs-issues
Closed

fix(jindo): make async reads concurrency-safe#426
zjw1111 wants to merge 4 commits into
alibaba:mainfrom
zjw1111:codex/reproduce-jindo-fs-issues

Conversation

@zjw1111

@zjw1111 zjw1111 commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Purpose

Linked issue: N/A

JindoInputStream reused one member std::string_view across synchronous and concurrent asynchronous reads, and ignored failures returned by perform(). This could race between reads or leave a caller's completion callback unresolved.

This change:

  • gives every asynchronous read an independent result and completion state
  • completes the callback exactly once, including when perform() fails
  • keeps synchronous read results local and documents the caller-side async lifetime contract
  • adds real OSS coverage for concurrent sync/async reads and listing across an OSS page boundary
  • introduces the opt-in PAIMON_ENABLE_JINDO_TESTS switch and enables Jindo cases in the existing filesystem and write/read integration test binaries

Tests

  • cmake --build build --target paimon-fs-test -j64
  • cmake --build build --target paimon-write-and-read-inte-test -j64
  • Real OSS Jindo filesystem tests: 6 passed
  • FsType/FileSystemTest.TestFactoryCreator/1
  • FileFormatAndFileSystem/WriteAndReadInteTest.TestAppendSimple/1
  • pre-commit run --files ...
  • git diff --check

API and Format

Clarifies the existing InputStream::ReadAsync lifetime contract. No storage format or protocol changes.

Documentation

No user-facing documentation changes.

Generative AI tooling

Generated-by: OpenAI Codex (GPT-5)

Copilot AI review requested due to automatic review settings July 15, 2026 11:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR hardens the Jindo filesystem InputStream async read path against concurrency hazards by removing shared per-stream read state, ensuring callbacks complete exactly once (including on immediate perform() failure), and wiring opt-in remote OSS-backed Jindo tests into existing test binaries.

Changes:

  • Reworked JindoInputStream::ReadAsync to use per-request shared state and to surface/complete on perform() failures.
  • Removed reused std::string_view member from JindoInputStream; made sync reads use local result state and clarified InputStream::ReadAsync lifetime contract.
  • Added an opt-in PAIMON_ENABLE_JINDO_TESTS switch and enabled Jindo variants in existing fs + integration test targets, including new OSS pagination + concurrent read coverage.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
src/paimon/fs/jindo/jindo_file_system.h Removes shared std::string_view member used across reads.
src/paimon/fs/jindo/jindo_file_system.cpp Introduces per-async-call completion/result state and handles perform() failures.
include/paimon/fs/file_system.h Documents async read lifetime requirements for callers.
src/paimon/fs/jindo/jindo_file_system_test.cpp Adds OSS-backed tests for listing pagination and concurrent sync/async reads.
src/paimon/common/fs/file_system_test.cpp Enables Jindo test parameter under PAIMON_ENABLE_JINDO_TESTS.
test/inte/write_and_read_inte_test.cpp Enables Jindo integration parameter under PAIMON_ENABLE_JINDO_TESTS.
src/paimon/CMakeLists.txt Conditionally compiles/links Jindo fs tests and defines PAIMON_ENABLE_JINDO_TESTS.
test/inte/CMakeLists.txt Defines PAIMON_ENABLE_JINDO_TESTS for the integration test target when enabled.
CMakeLists.txt Adds PAIMON_ENABLE_JINDO_TESTS option and enforces prerequisite options.
cmake_modules/DefineOptions.cmake Exposes PAIMON_ENABLE_JINDO_TESTS as a configurable option.
CONTRIBUTING.md Adds Conventional Commits guidance and notes on avoiding force-push during review iteration.
AGENTS.md Adds repository-specific agent guidance (new file).
.github/PULL_REQUEST_TEMPLATE.md Updates PR title guidance to Conventional Commits format.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +19 to 21
#include <atomic>
#include <cassert>
#include <utility>
@zjw1111

zjw1111 commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator Author

Closing this PR because its head branch contained unrelated documentation commits. A clean replacement PR will follow from codex/fix-jindo-async-read.

@zjw1111 zjw1111 closed this Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants