From deaa652d3d543262f7ad0e712ee7bc48eaed7e14 Mon Sep 17 00:00:00 2001 From: jackylee Date: Wed, 15 Jul 2026 20:50:27 +0800 Subject: [PATCH] fix(spark): warn that overwrite destructively deletes existing files The overwrite branch in `VortexBatchWrite.createBatchWriterFactory` already deletes the existing files before writing new data, but it logged `overwrite currently does not do anything for vortex format`, which directly contradicts the delete on the line above and hides a genuinely dangerous operation: the delete is a pre-delete, and `abort()` only removes the newly written files, so it cannot restore what was deleted if the write later fails. Replace the misleading message with a WARN that states how many files under which path are being deleted and that the operation cannot be undone if the subsequent write fails. This also folds in the file count previously logged at INFO, so it is a single, accurate log line. No write/delete behavior changes. Signed-off-by: jackylee --- .../java/dev/vortex/spark/write/VortexBatchWrite.java | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/java/vortex-spark/src/main/java/dev/vortex/spark/write/VortexBatchWrite.java b/java/vortex-spark/src/main/java/dev/vortex/spark/write/VortexBatchWrite.java index 7f96bac4404..03a70c2f676 100644 --- a/java/vortex-spark/src/main/java/dev/vortex/spark/write/VortexBatchWrite.java +++ b/java/vortex-spark/src/main/java/dev/vortex/spark/write/VortexBatchWrite.java @@ -90,9 +90,15 @@ public DataWriterFactory createBatchWriterFactory(PhysicalWriteInfo info) { if (overwrite) { var session = VortexSparkSession.get(options); var uris = NativeFiles.listFiles(session, outputPath, options); - log.info("truncating table with {} files", uris.size()); + // Deleting the existing files is destructive and happens before the new data is written: + // if the subsequent write fails, abort() only removes the newly written files and cannot + // restore what was deleted here. Log loudly so operators can see what was removed. + log.warn( + "Deleting {} existing file(s) under {} because of overwrite, before writing new data; " + + "this cannot be undone if the subsequent write fails", + uris.size(), + outputPath); NativeFiles.delete(session, uris.toArray(new String[0]), options); - log.warn("overwrite currently does not do anything for vortex format"); } return new VortexDataWriterFactory(outputPath, schema, options, resolvedTransforms);