Commit 9ecfd40
authored
[fix](job) fix StreamingJob loadStatistic reset to zero after FE checkpoint restart (#61997)
### What problem does this PR solve?
Problem Summary:
1. When FE restarts from a checkpoint image, `replayOnCommitted()` is
not called
for transactions committed before the checkpoint. Since `jobStatistic`
in
`StreamingInsertJob` lacked `@SerializedName`, it was not written into
the
image, causing `scannedRows` and `loadBytes` to reset to zero after
every
checkpoint-based restart.
2. add restart case for cdc tvf
#### Root Cause
`jobStatistic` accumulates statistics via `+=` in `replayOnCommitted()`.
Unlike `offset` (which is an assignment and idempotent), accumulated
values
cannot be recovered from post-checkpoint txn replay alone. The field
must be
persisted in the image to survive checkpoint restarts.1 parent f5783c0 commit 9ecfd40
7 files changed
Lines changed: 313 additions & 25 deletions
File tree
- fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming
- regression-test
- data/job_p0/streaming_job/cdc/tvf
- suites/job_p0/streaming_job
- cdc/tvf
Lines changed: 2 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
108 | 108 | | |
109 | 109 | | |
110 | 110 | | |
| 111 | + | |
| 112 | + | |
111 | 113 | | |
112 | 114 | | |
113 | 115 | | |
| |||
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
9 | | - | |
| 9 | + | |
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
| |||
Lines changed: 12 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
Lines changed: 2 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
58 | 58 | | |
59 | 59 | | |
60 | 60 | | |
61 | | - | |
| 61 | + | |
62 | 62 | | |
63 | 63 | | |
64 | | - | |
| 64 | + | |
65 | 65 | | |
66 | 66 | | |
67 | 67 | | |
| |||
Lines changed: 7 additions & 20 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
30 | | - | |
| 30 | + | |
31 | 31 | | |
32 | 32 | | |
33 | 33 | | |
| |||
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
48 | | - | |
49 | 48 | | |
50 | 49 | | |
51 | | - | |
| 50 | + | |
52 | 51 | | |
53 | 52 | | |
54 | | - | |
| 53 | + | |
55 | 54 | | |
56 | 55 | | |
57 | 56 | | |
| |||
112 | 111 | | |
113 | 112 | | |
114 | 113 | | |
| 114 | + | |
| 115 | + | |
115 | 116 | | |
116 | 117 | | |
117 | 118 | | |
| |||
169 | 170 | | |
170 | 171 | | |
171 | 172 | | |
172 | | - | |
173 | 173 | | |
174 | 174 | | |
175 | | - | |
176 | | - | |
| 175 | + | |
| 176 | + | |
177 | 177 | | |
178 | 178 | | |
179 | 179 | | |
| |||
204 | 204 | | |
205 | 205 | | |
206 | 206 | | |
207 | | - | |
208 | | - | |
209 | | - | |
210 | | - | |
211 | | - | |
212 | | - | |
213 | | - | |
214 | | - | |
215 | | - | |
216 | | - | |
217 | | - | |
218 | | - | |
219 | | - | |
220 | 207 | | |
221 | 208 | | |
222 | 209 | | |
| |||
0 commit comments