Skip to content

Commit 1bf87e6

Browse files
authored
fix(spark): warn that overwrite destructively deletes existing files (#8767)
## Rationale for this change The overwrite branch in `VortexBatchWrite.createBatchWriterFactory` already deletes the existing files before writing new data, but it logged: ``` overwrite currently does not do anything for vortex format ``` This message directly contradicts the `NativeFiles.delete(...)` call on the line right above it, and it hides a genuinely dangerous operation. The delete is a *pre-delete* (it runs before any new data is written), and `abort()` only removes the newly written files — it cannot restore what was deleted here. So if the subsequent write fails, the old data is already gone and unrecoverable, yet the log claims overwrite "does not do anything". The message dates back to when overwrite genuinely was a no-op; the actual deletion was added later (New Java Scan API, #7527) without updating the warning. ## What changes are included in this PR? Replace the misleading `warn` with one that accurately describes the destructive action: ```java // Deleting the existing files is destructive and happens before the new data is written: // if the subsequent write fails, abort() only removes the newly written files and cannot // restore what was deleted here. Log loudly so operators can see what was removed. log.warn( "Deleting {} existing file(s) under {} because of overwrite, before writing new data; " + "this cannot be undone if the subsequent write fails", uris.size(), outputPath); ``` This keeps the WARN level (deleting existing data warrants it), folds in the file count that was previously logged separately at INFO, and adds a short code comment documenting the pre-delete / non-atomic behavior for future maintainers. There is **no change to write or delete behavior** — this is purely a logging/clarity fix. ## What APIs are changed? Are there any user-facing changes? No API changes. The only user-facing change is the log output: operators will now see an accurate warning about the destructive deletion instead of a contradictory "does not do anything" message. Signed-off-by: jackylee <qcsd2011@gmail.com>
1 parent 204dacc commit 1bf87e6

1 file changed

Lines changed: 8 additions & 2 deletions

File tree

java/vortex-spark/src/main/java/dev/vortex/spark/write/VortexBatchWrite.java

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -90,9 +90,15 @@ public DataWriterFactory createBatchWriterFactory(PhysicalWriteInfo info) {
9090
if (overwrite) {
9191
var session = VortexSparkSession.get(options);
9292
var uris = NativeFiles.listFiles(session, outputPath, options);
93-
log.info("truncating table with {} files", uris.size());
93+
// Deleting the existing files is destructive and happens before the new data is written:
94+
// if the subsequent write fails, abort() only removes the newly written files and cannot
95+
// restore what was deleted here. Log loudly so operators can see what was removed.
96+
log.warn(
97+
"Deleting {} existing file(s) under {} because of overwrite, before writing new data; "
98+
+ "this cannot be undone if the subsequent write fails",
99+
uris.size(),
100+
outputPath);
94101
NativeFiles.delete(session, uris.toArray(new String[0]), options);
95-
log.warn("overwrite currently does not do anything for vortex format");
96102
}
97103

98104
return new VortexDataWriterFactory(outputPath, schema, options, resolvedTransforms);

0 commit comments

Comments
 (0)