Skip to content

Commit deaa652

Browse files
committed
fix(spark): warn that overwrite destructively deletes existing files
The overwrite branch in `VortexBatchWrite.createBatchWriterFactory` already deletes the existing files before writing new data, but it logged `overwrite currently does not do anything for vortex format`, which directly contradicts the delete on the line above and hides a genuinely dangerous operation: the delete is a pre-delete, and `abort()` only removes the newly written files, so it cannot restore what was deleted if the write later fails. Replace the misleading message with a WARN that states how many files under which path are being deleted and that the operation cannot be undone if the subsequent write fails. This also folds in the file count previously logged at INFO, so it is a single, accurate log line. No write/delete behavior changes. Signed-off-by: jackylee <qcsd2011@gmail.com>
1 parent 369fd32 commit deaa652

1 file changed

Lines changed: 8 additions & 2 deletions

File tree

java/vortex-spark/src/main/java/dev/vortex/spark/write/VortexBatchWrite.java

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -90,9 +90,15 @@ public DataWriterFactory createBatchWriterFactory(PhysicalWriteInfo info) {
9090
if (overwrite) {
9191
var session = VortexSparkSession.get(options);
9292
var uris = NativeFiles.listFiles(session, outputPath, options);
93-
log.info("truncating table with {} files", uris.size());
93+
// Deleting the existing files is destructive and happens before the new data is written:
94+
// if the subsequent write fails, abort() only removes the newly written files and cannot
95+
// restore what was deleted here. Log loudly so operators can see what was removed.
96+
log.warn(
97+
"Deleting {} existing file(s) under {} because of overwrite, before writing new data; "
98+
+ "this cannot be undone if the subsequent write fails",
99+
uris.size(),
100+
outputPath);
94101
NativeFiles.delete(session, uris.toArray(new String[0]), options);
95-
log.warn("overwrite currently does not do anything for vortex format");
96102
}
97103

98104
return new VortexDataWriterFactory(outputPath, schema, options, resolvedTransforms);

0 commit comments

Comments
 (0)