Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -239,13 +239,17 @@ public UpdateNamespacePropertiesResponse updateNamespaceProperties(
Map<String, String> startProperties = catalog.loadNamespaceMetadata(namespace);
Set<String> missing = Sets.difference(removals, startProperties.keySet());

if (!updates.isEmpty()) {
catalog.setProperties(namespace, updates);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean Polaris propagates namespace property updates to Federated catalogs even if CatalogEntity.isStaticFacade() is true? @dennishuo WDYT?

}
if (catalog instanceof IcebergCatalog polarisCatalog) {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Due to that we have a direct dependency with IcebergCatalog.
As CatalogHandlerUtils implements the SupportNamespace, it should not be "linked" to any catalog specific implementation.

I think it should be cleaner to isolate this in a specific method (for clarity).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, having instanceof checks in this case does not look nice from the design POV.

If we have to have custom handling for the internal catalog (which is ok), perhaps it is time to refactor this code to use proper polymorphic catalog objects and allow implementations to decide how to best handle updates.

Federated catalogs will have to delegate to the appropriate API.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed with @jbonofre and @dimas-b, please consider replacing catalog instanceof IcebergCatalog with a dedicated capability interface.

polarisCatalog.updateProperties(namespace, removals, updates);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I think if both updates and removals are empty, there's no need to call updateProperties no?

} else {
if (!updates.isEmpty()) {
catalog.setProperties(namespace, updates);
}

if (!removals.isEmpty()) {
// remove the original set just in case there was an update just after loading properties
catalog.removeProperties(namespace, removals);
if (!removals.isEmpty()) {
// remove the original set just in case there was an update just after loading properties
catalog.removeProperties(namespace, removals);
}
}

return UpdateNamespacePropertiesResponse.builder()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -735,6 +735,63 @@ public boolean setProperties(Namespace namespace, Map<String, String> properties
return true;
}

/**
* Update properties for a namespace.
*
* @param namespace the namespace to update properties for
* @param removals the set of properties to remove
* @param updates the map of properties to update
* @return true if the properties were updated, false otherwise
*/
public boolean updateProperties(
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The returned boolean is never used / checked.

Namespace namespace, Set<String> removals, Map<String, String> updates)
throws NoSuchNamespaceException {
PolarisResolvedPathWrapper resolvedEntities = resolvedEntityView.getResolvedPath(namespace);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not using resolvedEntityView.getResolvedPath(ResolvedPathKey.ofNamespace(namespace)); like in other methods here ?

if (resolvedEntities == null) {
throw noSuchNamespaceException(namespace);
}
PolarisEntity entity = resolvedEntities.getRawLeafEntity();
Map<String, String> newProperties = new HashMap<>(entity.getPropertiesAsMap());

// Merge new properties into existing map.
newProperties.putAll(updates);
// Remove properties.
removals.forEach(newProperties::remove);

PolarisEntity updatedEntity =
new PolarisEntity.Builder(entity).setProperties(newProperties).build();

if (!realmConfig.getConfig(FeatureConfiguration.ALLOW_NAMESPACE_LOCATION_OVERLAP)) {
LOGGER.debug("Validating no overlap with sibling tables or namespaces");
validateNoLocationOverlap(
NamespaceEntity.of(updatedEntity), resolvedEntities.getRawParentPath());
} else {
LOGGER.debug("Skipping location overlap validation for namespace '{}'", namespace);
}
if (!realmConfig.getConfig(
BehaviorChangeConfiguration.ALLOW_NAMESPACE_CUSTOM_LOCATION, catalogEntity)) {
if (updates.containsKey(PolarisEntityConstants.ENTITY_BASE_LOCATION)) {
validateNamespaceLocation(NamespaceEntity.of(entity), resolvedEntities);
}
}
Comment on lines +764 to +776
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These checks appear twice, can we extract a common utility method?


List<PolarisEntity> parentPath = resolvedEntities.getRawFullPath();
PolarisEntity returnedEntity =
Optional.ofNullable(
getMetaStoreManager()
.updateEntityPropertiesIfNotChanged(
getCurrentPolarisContext(),
PolarisEntity.toCoreList(parentPath),
updatedEntity)
.getEntity())
.map(PolarisEntity::new)
.orElse(null);
if (returnedEntity == null) {
throw new CommitConflictException("Concurrent modification of namespace: %s", namespace);
}
return true;
}

@Override
public boolean removeProperties(Namespace namespace, Set<String> properties)
throws NoSuchNamespaceException {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@
import com.azure.core.exception.HttpResponseException;
import com.google.cloud.storage.StorageException;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Iterables;
import com.google.common.collect.Streams;
import io.quarkus.test.junit.QuarkusMock;
Expand Down Expand Up @@ -2614,4 +2615,30 @@ public void testPaginatedListNamespaces() {
}
}
}

@Test
public void testUpdateNamespacePropertiesAtomic() {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be preferable to promote this test to PolarisRestCatalogIntegrationBase, which acts almost like a TCK for Polaris. WDYT?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What in this test ensures that the operation is "atomic"? 🤔

catalog.createNamespace(
NS,
ImmutableMap.of(
"prop1", "val1",
"prop2", "val2",
"prop3", "val3"));

catalog.updateProperties(
NS,
ImmutableSet.of("prop1", "prop2", "missing_prop"),
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: let's keep one old prop to assert that it is preserved after the update:

Suggested change
ImmutableSet.of("prop1", "prop2", "missing_prop"),
ImmutableSet.of("prop1", "missing_prop"),

ImmutableMap.of(
"prop3", "new_val3",
"prop4", "val4",
"prop5", "val5"));

Map<String, String> properties = catalog.loadNamespaceMetadata(NS);
Assertions.assertThat(properties).containsEntry("prop3", "new_val3");
Assertions.assertThat(properties).containsEntry("prop4", "val4");
Assertions.assertThat(properties).containsEntry("prop5", "val5");
Assertions.assertThat(properties).doesNotContainKey("prop1");
Assertions.assertThat(properties).doesNotContainKey("prop2");
}
}

Loading