Skip to content

Location overlap checks are not resilient to deleting sibling entities concurrently #4407

Closed
#4408
Closed
Location overlap checks are not resilient to deleting sibling entities concurrently#4407
#4408
Assignees
dimas-b
Labels
bugSomething isn't working
@dimas-b

Description

@dimas-b
dimas-b
opened on May 11, 2026
  1. Start Polaris Server (any Persistence).
  2. Create catalog (any storage).
  3. Grant catalog access to the admin role (see below)
  4. Create namespace ns
  5. Run two parallel processes each running in a loop: a) create / drop namespace ns.n1 b) perform a staged table create for table ts1 in namespace ns
  6. Observe that after a while process b fails with a 500 response from Polaris.

Example scripts:

Run with in-memory persistence:

env POLARIS_BOOTSTRAP_CREDENTIALS=POLARIS,pol123,pol123s java -jar runtime/server/build/quarkus-app/quarkus-run.jar

Create RustFS catalog:

./polaris --client-id pol123 --client-secret pol123s catalogs create polaris --storage-type S3 --default-base-location 's3://pol/' --endpoint http://rustfs.local:9000 --path-style-access

Grant catalog access to the admin role:

./polaris --realm POLARIS --client-id pol123 --client-secret pol123s privileges catalog grant --catalog polaris --catalog-role catalog_admin CATALOG_MANAGE_CONTENT

Create first namespace (ns) (fish shell):

$ set -x POLARIS_TOKEN (curl -X POST http://localhost:8181/api/catalog/v1/oauth/tokens --data-urlencode "scope=PRINCIPAL_ROLE:ALL" --data-urlencode "grant_type=client_credentials" --data-urlencode "client_id=pol123" --data-urlencode "client_secret=pol123s"|jq -r .access_token)

$ curl -H "Authorization: Bearer $POLARIS_TOKEN" -H 'Content-Type: application/json' -X POST 'http://localhost:8181/api/catalog/v1/polaris/namespaces/' -d '{"namespace":["ns"]}'

Process A (fish shell):

$ set -x POLARIS_TOKEN (curl -X POST http://localhost:8181/api/catalog/v1/oauth/tokens --data-urlencode "scope=PRINCIPAL_ROLE:ALL" --data-urlencode "grant_type=client_credentials" --data-urlencode "client_id=pol123" --data-urlencode "client_secret=pol123s"|jq -r .access_token)

$ for n in (seq 1 5000) ; curl -H "Authorization: Bearer $POLARIS_TOKEN" -H 'Content-Type: application/json' -X POST 'http://localhost:8181/api/catalog/v1/polaris/namespaces/' -d '{"namespace":["ns", "ns1"]}' ; curl -H "Authorization: Bearer $POLARIS_TOKEN" -H 'Content-Type: application/json' -X DELETE 'http://localhost:8181/api/catalog/v1/polaris/namespaces/ns%1fns1' ; end

Process B (fish shell):

$ set -x POLARIS_TOKEN (curl -X POST http://localhost:8181/api/catalog/v1/oauth/tokens --data-urlencode "scope=PRINCIPAL_ROLE:ALL" --data-urlencode "grant_type=client_credentials" --data-urlencode "client_id=pol123" --data-urlencode "client_secret=pol123s"|jq -r .access_token)

$ for n in (seq 1 5000) ; set v (curl -H "Authorization: Bearer $POLARIS_TOKEN" -H  'X-Iceberg-Access-Delegation: vended-credentials' -H "Content-Type: application/json" -X POST 'http://localhost:8181/api/catalog/v1/polaris/namespaces/ns/tables' -d '{
        "name": "ts1",
        "stage-create": true,
        "schema": {
          "type": "struct",
          "schema-id": 0,
          "fields": [
            { "id": 1, "name": "id",    "required": true,  "type": "long" },
            { "id": 2, "name": "name",  "required": false, "type": "string" },
            { "id": 3, "name": "ts",    "required": false, "type": "timestamptz" }
          ]
        },
        "partition-spec": {
          "spec-id": 0,
          "fields": []
        },
        "write-order": {
          "order-id": 0,
          "fields": []
        },
        "properties": {
          "write.format.default": "parquet"
        }
      }' -s -w "%{http_code}" -o /dev/null) ; echo $v ; if test $v != "200" ; break; end; end

Observed Polaris exception:

2026-05-11 18:59:51,245 INFO  [io.qua.htt.access-log] [55ee3dea-52d6-4614-a87f-a70bcfda0be0_0000000000000017029,POLARIS] [,,,] (executor-thread-5) 0:0:0:0:0:0:0:1 - root [11/May/2026:18:59:51 -0400] "DELETE /api/catalog/v1/polaris/namespaces/ns%1fns1 HTTP/1.1" 204 -
2026-05-11 18:59:51,246 INFO  [org.apa.pol.ser.exc.IcebergExceptionMapper] [55ee3dea-52d6-4614-a87f-a70bcfda0be0_0000000000000017028,POLARIS] [,,,] (executor-thread-4) Handling runtimeException Unable to resolve sibling entities to validate location - PATH_COULD_NOT_BE_FULLY_RESOLVED
2026-05-11 18:59:51,246 ERROR [org.apa.pol.ser.exc.IcebergExceptionMapper] [55ee3dea-52d6-4614-a87f-a70bcfda0be0_0000000000000017028,POLARIS] [,,,] (executor-thread-4) Unhandled exception returning INTERNAL_SERVER_ERROR: java.lang.IllegalStateException: Unable to resolve sibling entities to validate location - PATH_COULD_NOT_BE_FULLY_RESOLVED
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalog.validateNoLocationOverlap(IcebergCatalog.java:1272)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalog.validateNoLocationOverlap(IcebergCatalog.java:1085)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalog.lambda$validateStagedTableCreate$7(IcebergCatalog.java:1015)
	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalog.validateStagedTableCreate(IcebergCatalog.java:1013)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogHandler.createTableStaged(IcebergCatalogHandler.java:597)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.lambda$createTable$7(IcebergCatalogAdapter.java:284)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.withCatalogByName(IcebergCatalogAdapter.java:112)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.withCatalog(IcebergCatalogAdapter.java:104)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.createTable(IcebergCatalogAdapter.java:273)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter_Subclass.createTable$$superforward(Unknown Source)
	at org.apache.polaris.service.catalog.iceberg.IcebergRestCatalogEventServiceDelegator_Gj_WCptqTcdHu-fbZfgVkAwPXCI_Delegate_Subclass.createTable(Unknown Source)
	at org.apache.polaris.service.catalog.iceberg.IcebergRestCatalogEventServiceDelegator.createTable(IcebergRestCatalogEventServiceDelegator.java:307)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter_Subclass.createTable(Unknown Source)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter_ClientProxy.createTable(Unknown Source)
	at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi.createTable(IcebergRestCatalogApi.java:193)
	at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass.createTable$$superforward(Unknown Source)
	at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass$2.apply(Unknown Source)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)
	at io.smallrye.faulttolerance.FaultToleranceInterceptor.lambda$syncFlow$8(FaultToleranceInterceptor.java:364)
	at io.smallrye.faulttolerance.core.Future.from(Future.java:85)
	at io.smallrye.faulttolerance.FaultToleranceInterceptor.lambda$syncFlow$9(FaultToleranceInterceptor.java:364)
	at io.smallrye.faulttolerance.core.FaultToleranceContext.call(FaultToleranceContext.java:20)
	at io.smallrye.faulttolerance.core.Invocation.apply(Invocation.java:29)
	at io.smallrye.faulttolerance.core.metrics.MetricsCollector.apply(MetricsCollector.java:98)
	at io.smallrye.faulttolerance.FaultToleranceInterceptor.syncFlow(FaultToleranceInterceptor.java:367)
	at io.smallrye.faulttolerance.FaultToleranceInterceptor.intercept(FaultToleranceInterceptor.java:205)
	at io.smallrye.faulttolerance.FaultToleranceInterceptor_Bean.intercept(Unknown Source)
	at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)
	at io.quarkus.micrometer.runtime.MicrometerTimedInterceptor.timedMethod(MicrometerTimedInterceptor.java:79)
	at io.quarkus.micrometer.runtime.MicrometerTimedInterceptor_Bean.intercept(Unknown Source)
	at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)
	at io.quarkus.security.runtime.interceptor.SecurityHandler.handle(SecurityHandler.java:27)
	at io.quarkus.security.runtime.interceptor.RolesAllowedInterceptor.intercept(RolesAllowedInterceptor.java:31)
	at io.quarkus.security.runtime.interceptor.RolesAllowedInterceptor_Bean.intercept(Unknown Source)
	at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62)
	at io.quarkus.resteasy.reactive.server.runtime.StandardSecurityCheckInterceptor.intercept(StandardSecurityCheckInterceptor.java:48)
	at io.quarkus.resteasy.reactive.server.runtime.StandardSecurityCheckInterceptor$RolesAllowedInterceptor_Bean.intercept(Unknown Source)
	at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30)
	at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27)
	at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass.createTable(Unknown Source)
	at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi$quarkusrestinvoker$createTable_01f5a1bd6d7815fd3314a553161c943c8cd03101.invoke(Unknown Source)
	at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29)
	at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:190)
	at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147)
	at io.quarkus.vertx.core.runtime.VertxCoreRecorder$15.runWith(VertxCoreRecorder.java:677)
	at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2651)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2630)
	at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1622)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1589)
	at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
	at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions