Commit deaa652
committed
fix(spark): warn that overwrite destructively deletes existing files
The overwrite branch in `VortexBatchWrite.createBatchWriterFactory` already
deletes the existing files before writing new data, but it logged
`overwrite currently does not do anything for vortex format`, which directly
contradicts the delete on the line above and hides a genuinely dangerous
operation: the delete is a pre-delete, and `abort()` only removes the newly
written files, so it cannot restore what was deleted if the write later fails.
Replace the misleading message with a WARN that states how many files under
which path are being deleted and that the operation cannot be undone if the
subsequent write fails. This also folds in the file count previously logged at
INFO, so it is a single, accurate log line. No write/delete behavior changes.
Signed-off-by: jackylee <qcsd2011@gmail.com>1 parent 369fd32 commit deaa652
1 file changed
Lines changed: 8 additions & 2 deletions
Lines changed: 8 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
90 | 90 | | |
91 | 91 | | |
92 | 92 | | |
93 | | - | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
94 | 101 | | |
95 | | - | |
96 | 102 | | |
97 | 103 | | |
98 | 104 | | |
| |||
0 commit comments