fix(spark): warn that overwrite destructively deletes existing files#8767
Merged
robert3005 merged 1 commit intoJul 15, 2026
Merged
Conversation
The overwrite branch in `VortexBatchWrite.createBatchWriterFactory` already deletes the existing files before writing new data, but it logged `overwrite currently does not do anything for vortex format`, which directly contradicts the delete on the line above and hides a genuinely dangerous operation: the delete is a pre-delete, and `abort()` only removes the newly written files, so it cannot restore what was deleted if the write later fails. Replace the misleading message with a WARN that states how many files under which path are being deleted and that the operation cannot be undone if the subsequent write fails. This also folds in the file count previously logged at INFO, so it is a single, accurate log line. No write/delete behavior changes. Signed-off-by: jackylee <qcsd2011@gmail.com>
Contributor
Author
|
Could a maintainer please add the |
robert3005
approved these changes
Jul 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
The overwrite branch in
VortexBatchWrite.createBatchWriterFactoryalready deletesthe existing files before writing new data, but it logged:
This message directly contradicts the
NativeFiles.delete(...)call on the line rightabove it, and it hides a genuinely dangerous operation. The delete is a pre-delete
(it runs before any new data is written), and
abort()only removes the newly writtenfiles — it cannot restore what was deleted here. So if the subsequent write fails, the
old data is already gone and unrecoverable, yet the log claims overwrite "does not do
anything".
The message dates back to when overwrite genuinely was a no-op; the actual deletion was
added later (New Java Scan API, #7527) without updating the warning.
What changes are included in this PR?
Replace the misleading
warnwith one that accurately describes the destructive action:This keeps the WARN level (deleting existing data warrants it), folds in the file count
that was previously logged separately at INFO, and adds a short code comment documenting
the pre-delete / non-atomic behavior for future maintainers.
There is no change to write or delete behavior — this is purely a logging/clarity fix.
What APIs are changed? Are there any user-facing changes?
No API changes. The only user-facing change is the log output: operators will now see an
accurate warning about the destructive deletion instead of a contradictory "does not do
anything" message.