Skip to content

fix(spark): warn that overwrite destructively deletes existing files#8767

Merged
robert3005 merged 1 commit into
vortex-data:developfrom
jackylee-ch:spark-overwrite-warn-destructive-delete
Jul 15, 2026
Merged

fix(spark): warn that overwrite destructively deletes existing files#8767
robert3005 merged 1 commit into
vortex-data:developfrom
jackylee-ch:spark-overwrite-warn-destructive-delete

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

Rationale for this change

The overwrite branch in VortexBatchWrite.createBatchWriterFactory already deletes
the existing files before writing new data, but it logged:

overwrite currently does not do anything for vortex format

This message directly contradicts the NativeFiles.delete(...) call on the line right
above it, and it hides a genuinely dangerous operation. The delete is a pre-delete
(it runs before any new data is written), and abort() only removes the newly written
files — it cannot restore what was deleted here. So if the subsequent write fails, the
old data is already gone and unrecoverable, yet the log claims overwrite "does not do
anything".

The message dates back to when overwrite genuinely was a no-op; the actual deletion was
added later (New Java Scan API, #7527) without updating the warning.

What changes are included in this PR?

Replace the misleading warn with one that accurately describes the destructive action:

// Deleting the existing files is destructive and happens before the new data is written:
// if the subsequent write fails, abort() only removes the newly written files and cannot
// restore what was deleted here. Log loudly so operators can see what was removed.
log.warn(
        "Deleting {} existing file(s) under {} because of overwrite, before writing new data; "
                + "this cannot be undone if the subsequent write fails",
        uris.size(),
        outputPath);

This keeps the WARN level (deleting existing data warrants it), folds in the file count
that was previously logged separately at INFO, and adds a short code comment documenting
the pre-delete / non-atomic behavior for future maintainers.

There is no change to write or delete behavior — this is purely a logging/clarity fix.

What APIs are changed? Are there any user-facing changes?

No API changes. The only user-facing change is the log output: operators will now see an
accurate warning about the destructive deletion instead of a contradictory "does not do
anything" message.

The overwrite branch in `VortexBatchWrite.createBatchWriterFactory` already
deletes the existing files before writing new data, but it logged
`overwrite currently does not do anything for vortex format`, which directly
contradicts the delete on the line above and hides a genuinely dangerous
operation: the delete is a pre-delete, and `abort()` only removes the newly
written files, so it cannot restore what was deleted if the write later fails.

Replace the misleading message with a WARN that states how many files under
which path are being deleted and that the operation cannot be undone if the
subsequent write fails. This also folds in the file count previously logged at
INFO, so it is a single, accurate log line. No write/delete behavior changes.

Signed-off-by: jackylee <qcsd2011@gmail.com>
@jackylee-ch

Copy link
Copy Markdown
Contributor Author

Could a maintainer please add the changelog/fix label? I don't have permission to set labels on the repo, and the labels check requires exactly one changelog/* label. Thanks!

@robert3005 robert3005 added the changelog/fix A bug fix label Jul 15, 2026
@robert3005 robert3005 merged commit 1bf87e6 into vortex-data:develop Jul 15, 2026
81 of 82 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants