Spark: block DROP TABLE when the user cannot delete the table data on HDFS (1.11.0)

[anandnalya]

What

Adds a pre-drop HDFS permission check so that DROP TABLE and DROP TABLE PURGE fail up front when the current user cannot delete the table's files. Stacked on #2 (base branch feat/spark-location-allowlist-1.11.0).

Why

With a Hive catalog, DROP TABLE removes the metastore entry without deleting data, and DROP TABLE PURGE removes the metastore entry before deleting files client-side. In both cases a user without HDFS delete permission on the table location ends up with orphaned data that is no longer reachable through any catalog. Failing the drop keeps the catalog entry and the data consistent.

How

• New DropTablePermissionValidator calls FileSystem.access() — on HDFS a NameNode-side checkAccess RPC evaluated against the caller's UGI (groups and ACLs included) — requiring WRITE+EXECUTE on the table directory and on its parent (both are needed to delete the tree and unlink the directory).
• Sticky-bit protection is evaluated the way the NameNode's FSPermissionChecker does on delete: when a directory has the sticky bit set, an entry can only be removed by the entry's owner or the directory's owner. The rule is applied to the parent directory (unlinking the table directory) and, when the table directory itself is sticky and not owned by the caller, to each of its immediate children.
• Hooked into SparkCatalog#dropTableWithoutPurging, which both dropTable and purgeTable route through, so plain drops, purges, and SparkSessionCatalog delegation are all covered.
• Only hdfs locations are enforced; other filesystems (s3a, file) fall back to a non-authoritative client-side check and are skipped.
• Null/empty/missing locations and tables with unreadable metadata pass through, so dangling metastore entries can still be cleaned up.
• Service accounts (usernames starting with bot-) are exempt from the check entirely.
• Applied to spark/v3.4, v3.5, v4.0, v4.1.

Limitations

• The check inspects the top level only (table dir, its parent, and immediate children for the sticky case), not the full tree; a deeper subtree with divergent ownership can still fail mid-purge. Iceberg-written trees have uniform ownership in practice.
• purgeTable still drops the metastore entry before deleting files; the pre-check shrinks but does not eliminate the orphaning window.
• HDFS superusers bypass sticky-bit checks server-side but cannot be detected client-side, so a superuser may be blocked spuriously when sticky-protected entries are owned by other users.

Testing

TestDropTablePermissionValidator (15 cases: allow, deny on table dir, deny on parent, non-HDFS skip, missing location, concurrent delete race, RPC failure, null/empty location, six sticky-bit scenarios covering parent-sticky and table-dir-sticky with owner/non-owner combinations, and the bot- exemption via UserGroupInformation.doAs) passes on all four Spark versions; checkstyle/spotless clean.

[Copilot]

Pull request overview

This PR adds a Spark-side pre-drop filesystem permission validator to prevent DROP TABLE / DROP TABLE PURGE from orphaning HDFS data when the current user cannot delete the table’s directory. The change is wired into SparkCatalog#dropTableWithoutPurging across Spark 3.4, 3.5, 4.0, and 4.1, and includes unit tests for key allow/deny and sticky-bit scenarios.

Changes:

• Introduce DropTablePermissionValidator to perform HDFS FileSystem.access() checks (plus sticky-bit ownership rules) before drop/purge.
• Invoke the validator from SparkCatalog#dropTableWithoutPurging (both path identifiers and catalog-backed identifiers).
• Add TestDropTablePermissionValidator coverage across all supported Spark versions.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java	Calls pre-drop validator and stores Hadoop Configuration for reuse.
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/DropTablePermissionValidator.java	New HDFS permission/sticky-bit validator implementation.
spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/TestDropTablePermissionValidator.java	Unit tests for validator behavior (including sticky bit and bot exemption).
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java	Calls pre-drop validator and stores Hadoop Configuration for reuse.
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/DropTablePermissionValidator.java	New HDFS permission/sticky-bit validator implementation.
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestDropTablePermissionValidator.java	Unit tests for validator behavior (including sticky bit and bot exemption).
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java	Calls pre-drop validator and stores Hadoop Configuration for reuse.
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/DropTablePermissionValidator.java	New HDFS permission/sticky-bit validator implementation.
spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/TestDropTablePermissionValidator.java	Unit tests for validator behavior (including sticky bit and bot exemption).
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java	Calls pre-drop validator and stores Hadoop Configuration for reuse.
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/DropTablePermissionValidator.java	New HDFS permission/sticky-bit validator implementation.
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/TestDropTablePermissionValidator.java	Unit tests for validator behavior (including sticky bit and bot exemption).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.