Skip to content

Commit 13e2c06

Browse files
HaHaJeffHaHaJeff
committed
docs: propose writer-local read-latest support
1 parent 4c2277c commit 13e2c06

1 file changed

Lines changed: 226 additions & 0 deletions

File tree

‎docs/proposals/writer-read-latest.md‎

Lines changed: 226 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,226 @@
1+
<!---
2+
Copyright 2026-present Alibaba Inc.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
-->
16+
17+
# [Proposal] Add Writer-local ReadLatest Support to Paimon C++
18+
19+
## 1. Motivation
20+
21+
Paimon C++ writers already keep intermediate state before committing a snapshot. Records may stay
22+
in the active write buffer, be spilled into sorted runs, or be written as prepared files before the
23+
final snapshot commit.
24+
25+
For embedded C++ applications, it is useful to read the latest state accepted by a writer before
26+
the snapshot is committed. Typical examples include validation and read-after-write flows owned by
27+
the same process. Today, applications that need this behavior usually maintain an external side
28+
buffer and duplicate merge logic outside Paimon.
29+
30+
That approach becomes hard to keep correct once the writer spills data or prepares files. The
31+
application has to know which data is still in memory, which data has been spilled, which data has
32+
been prepared, and how to merge those records with committed snapshot files. This duplicates
33+
behavior that already belongs to Paimon.
34+
35+
The proposed change exposes this state through a controlled writer-local read API, while preserving
36+
Paimon's normal snapshot isolation for committed readers.
37+
38+
## 2. Scope
39+
40+
This proposal targets **primary-key tables** in the first phase.
41+
42+
`ReadLatest` depends on primary-key merge semantics: same-key conflict resolution, sequence
43+
ordering, and delete handling. Append-only tables have different semantics, where repeated records
44+
remain independent appended rows rather than versions of the same key. Append-only table serving
45+
is therefore out of scope for the first phase.
46+
47+
The first phase focuses on correctness and API shape. Recovery from external write-ahead logs and
48+
serving-oriented performance optimization are left as follow-up work.
49+
50+
## 3. Proposed Change
51+
52+
Add a writer-local read path that can create a stable read view from:
53+
54+
- committed snapshot files visible to the writer;
55+
- active in-memory buffer owned by the writer;
56+
- spilled sorted runs owned by the writer;
57+
- prepared files created by the writer before snapshot commit.
58+
59+
The reader returns the latest primary-key result according to Paimon's existing primary-key merge
60+
semantics.
61+
62+
```mermaid
63+
flowchart LR
64+
subgraph "Committed State"
65+
Snapshot[Committed Snapshot]
66+
Files[Committed Data Files]
67+
Snapshot --> Files
68+
end
69+
70+
subgraph "Writer-local State"
71+
Buffer[Active Buffer]
72+
Spill[Spilled Runs]
73+
Prepared[Prepared Files]
74+
end
75+
76+
Files --> View[ReadLatest View]
77+
Buffer --> View
78+
Spill --> View
79+
Prepared --> View
80+
View --> Merge[Primary-key Merge Reader]
81+
Merge --> Result[Latest Records]
82+
```
83+
84+
## 4. Public API
85+
86+
The API should be explicit about the visibility model. The proposed naming is `ReadLatest`, not
87+
`ReadUncommitted`, because the feature does not introduce a table-wide isolation level. The names
88+
in the following example are proposed public API and are not present in the current public headers.
89+
90+
```cpp
91+
ReadLatestOptions options;
92+
std::unique_ptr<RecordReader> reader =
93+
writer->CreateReadLatestReader(options);
94+
```
95+
96+
Expected behavior:
97+
98+
- the API is called on a writer instance;
99+
- it includes data accepted by that writer;
100+
- it does not expose data from other writers;
101+
- it does not make uncommitted data globally visible;
102+
- it does not change existing snapshot readers.
103+
104+
The first implementation is intended to expose a full writer-local read view. Projection and
105+
predicate support should only be added where they can follow existing Paimon scan semantics safely.
106+
107+
## 5. Prototype
108+
109+
A local prototype has been implemented to validate the proposed direction in Paimon C++.
110+
111+
The prototype currently validates fixed-bucket primary-key tables with the deduplicate merge
112+
engine. Deletion vectors, predicates, and other merge engines are outside the validated prototype
113+
surface.
114+
115+
The prototype demonstrates:
116+
117+
- writer-local `ReadLatest` reader for primary-key tables;
118+
- reading records from the active write buffer before snapshot commit;
119+
- reading spilled writer-local sorted runs;
120+
- reading prepared files before snapshot commit;
121+
- merging committed snapshot files with writer-local records;
122+
- correct same-key insert, update, and delete resolution across snapshot files, memory buffer,
123+
spilled runs, and prepared files;
124+
- stable read view lifetime after later writer operations such as spill, prepare, commit, and close;
125+
- existing snapshot read and commit paths remain unchanged;
126+
- focused tests covering same-key mutations, multiple spills, prepared files, snapshot merge, delete
127+
handling, read-view lifetime, and failure cleanup.
128+
129+
The prototype is intended to validate API shape and correctness first. It does not claim to solve
130+
crash recovery from external write-ahead logs or serving-oriented performance optimization in the
131+
first phase.
132+
133+
## 6. Semantics
134+
135+
### 6.1 Visibility
136+
137+
`ReadLatest` is writer-local. It observes committed snapshot state plus uncommitted state owned by
138+
the same writer at the time the read view is created.
139+
140+
It is not a global read-uncommitted mode. Other readers continue to see committed snapshots only.
141+
142+
### 6.2 Read View Stability
143+
144+
Once `CreateReadLatestReader()` succeeds, the returned reader must remain readable until it is
145+
destroyed.
146+
147+
Later writer operations may continue:
148+
149+
- write more records;
150+
- spill buffered records;
151+
- prepare commit files;
152+
- commit a snapshot;
153+
- flush or close writer-owned resources.
154+
155+
Those later operations must not invalidate resources already pinned by the read view.
156+
157+
This proposal does not change the writer's existing thread-safety contract. It only requires that
158+
once a read view is created successfully, subsequent writer lifecycle operations must not make the
159+
returned reader dangling.
160+
161+
```mermaid
162+
sequenceDiagram
163+
participant App
164+
participant Writer
165+
participant View as ReadLatest View
166+
participant Reader
167+
168+
App->>Writer: Write records
169+
App->>Writer: CreateReadLatestReader()
170+
Writer->>View: Pin committed files and writer-local sources
171+
View->>Reader: Return stable reader
172+
App->>Writer: Continue write / spill / prepare / close
173+
App->>Reader: Consume pinned view safely
174+
```
175+
176+
### 6.3 Primary-key Result
177+
178+
For primary-key tables, the reader returns the latest visible mutation for each key in the read
179+
view.
180+
181+
- If the newest visible mutation is an insert or update, the latest row is returned.
182+
- If the newest visible mutation is a delete, the key is not returned.
183+
- If the same key appears in committed files and writer-local state, the writer-local newer mutation
184+
wins according to sequence ordering.
185+
186+
All writer-local sources in a read view should use the same sequence ordering as the existing
187+
primary-key writer and merge path. The read path should not introduce an independent ordering rule.
188+
189+
### 6.4 Resource Lifetime
190+
191+
All resources captured by a read view must remain valid until the reader is released. This includes
192+
active buffer snapshots, spilled run readers, committed data files, and prepared files.
193+
194+
Prepared files included in a read view must be retained until the view is released, regardless of
195+
later commit, close, or cleanup operations.
196+
197+
## 7. Compatibility
198+
199+
This proposal is additive.
200+
201+
- Existing snapshot read behavior remains unchanged.
202+
- Existing write, prepare commit, and snapshot commit behavior remains unchanged.
203+
- Existing applications do not need to opt in.
204+
- Applications that need writer-local latest reads can use the new writer API.
205+
206+
The implementation should not introduce additional overhead to the default snapshot read path.
207+
208+
## 8. Implementation Outline
209+
210+
If the community agrees with the direction, the implementation can be submitted incrementally:
211+
212+
1. Expose writer-owned buffered records as an internal readable source.
213+
2. Add readable sources for spilled writer-local sorted runs.
214+
3. Include prepared files in the writer-local read view before snapshot commit.
215+
4. Build a merge reader over committed files and writer-local sources.
216+
5. Pin resources through the lifetime of the returned reader.
217+
6. Add the public `CreateReadLatestReader()` writer API for a full writer-local read view.
218+
7. Add focused tests for same-key mutations, multiple spills, prepared files, snapshot merge, delete
219+
handling, read-view lifetime, and failure cleanup.
220+
221+
The implementation should reuse existing primary-key merge logic wherever possible and avoid
222+
introducing a second merge model outside the current table semantics.
223+
224+
## 9. Generative AI tooling
225+
226+
Generated-by: OpenAI Codex GPT-5

0 commit comments

Comments
 (0)