Optimize grant macros for performance and case-insensitive role matching

CHANGELOG.md

@@ -1,6 +1,19 @@
 # Data Engineers Snowflake DataOps Utils Project Changelog
 This file contains the changelog for the Data Engineers Snowflake DataOps Utils project, detailing updates, fixes, and enhancements made to the project over time.
 
+## v1.0.7 - 2026-06-25 - Grant Performance Optimization
+
+### Changed
+- Modified `grant_object` macro to be grant-only: it no longer revokes privileges from roles not in the supplied list or revokes non-desired privileges. The macro now only ensures the specified privileges are granted to the specified roles.
+- **Performance**: `grant_schema_read_specific` now checks existing privileges via `information_schema.object_privileges` and `SHOW FUTURE GRANTS` before issuing statements. Schemas where all grants are already in place are skipped entirely.
+- **Performance**: `grant_schema_monitor_specific` and `grant_schema_operate_specific` now skip roles that already have the required privilege and only issue `GRANT USAGE ON SCHEMA` when missing.
+- **Performance**: `grant_schema_procedure_usage_specific` replaced O(n×m) per-procedure `SHOW GRANTS` calls with a single `information_schema.object_privileges` query per schema.
+- **Performance**: `grant_schema_object_privileges` replaced per-object `SHOW GRANTS` calls with a single bulk `information_schema.object_privileges` query per schema.
+- **Performance**: `grant_internal_share_read` and `grant_external_share_read` now check existing share grants via `DESC SHARE` upfront and skip schemas/objects that are already granted.
+- **Performance**: `grant_share_read_specific_schema` now checks existing share state before issuing grants.
+- Added helper macros `_grants_get_schema_object_privs`, `_grants_get_schema_grants`, and `_grants_get_future_grants` to `_helpers.sql` for bulk privilege state checking.
+- Bumped version from 1.0.6 to 1.0.7
+
 ## v1.0.6 - 2026-06-23 - Database Clone Grant Ownership
 
 ### Added

README.md

@@ -2,7 +2,7 @@
 
 A macro-only [dbt](https://github.com/dbt-labs/dbt) package for Snowflake DataOps. Provides utilities for object lifecycle management, RBAC grant orchestration, dimensional modelling helpers, tagging, shares, and more.
 
-- **Version**: 1.0.6
+- **Version**: 1.0.7
 - **dbt**: `>=1.3.0, <3.0.0`
 - **Dependencies**: None (zero external package dependencies)
 - **dbt Fusion**: Compatible
@@ -108,7 +108,7 @@ The following `vars` can be set in your `dbt_project.yml` or via `--vars` on the
 | `grant_schema_procedure_usage(exclude_schemas, grant_roles)` | Grant USAGE on all procedures across all schemas |
 | `grant_schema_procedure_usage_specific(schemas, grant_roles, revoke_current_grants, dry_run)` | Grant USAGE on all procedures in specific schemas |
 | `grant_schema_object_privileges(object_type, schema_name, permissions, roles)` | Bulk grant privileges on all objects of a type within a schema |
-| `grant_object(object_type, objects, grant_types, grant_roles)` | Reconcile privilege sets on specific objects for roles |
+| `grant_object(object_type, objects, grant_types, grant_roles)` | Grant privileges on specific objects to roles (grant-only, no revokes) |
 | `grant_object_application(object_type, objects, grant_types, grant_applications)` | Reconcile privilege sets on specific objects for applications |
 | `grant_usage_to_application(object_type, prefix, grant_applications)` | Grant USAGE on objects matching a prefix to applications |
 | `grant_operate_to_application(prefix, grant_applications)` | Grant OPERATE on tasks matching a prefix to applications |
@@ -231,7 +231,7 @@ vars:
 | `grant_schema_operate*` | OPERATE on tasks/pipes + schema usage |
 | `grant_schema_procedure_usage*` | USAGE on all procedures + schema usage |
 | `grant_share_read*` | Secure view exposure to outbound shares |
-| `grant_object` | Per-object privilege reconciliation for roles |
+| `grant_object` | Per-object privilege granting for roles (no revokes) |
 | `grant_object_application` | Per-object privilege reconciliation for applications |
 | `grant_privileges` | Environment-aware bundle orchestrator |
 

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'dbt_dataengineers_utils'
-version: '1.0.6'
+version: '1.0.7'
 config-version: 2
 
 require-dbt-version: [">=1.9.4", "<3.0.0"]

integration_tests/macros/test_grants_helpers.sql

@@ -0,0 +1,113 @@
+{#
+  Test helper macros for grants management.
+  Run via: dbt run-operation test_grants_helpers
+  Returns nothing on success; raises an error on any assertion failure.
+#}
+{% macro test_grants_helpers() %}
+    {% set failures = [] %}
+
+    {# ── Test _grants_normalize_roles ── #}
+
+    {# Test 1: lowercase roles are uppercased #}
+    {% set result = dbt_dataengineers_utils._grants_normalize_roles(['analyst', 'developer']) %}
+    {% if result != ['ANALYST', 'DEVELOPER'] %}
+        {% do failures.append("Test 1 FAILED: lowercase normalization expected ['ANALYST', 'DEVELOPER'], got " ~ result) %}
+    {% endif %}
+
+    {# Test 2: mixed-case roles are uppercased #}
+    {% set result = dbt_dataengineers_utils._grants_normalize_roles(['Analyst', 'OPS_Support', 'readers_prod']) %}
+    {% if result != ['ANALYST', 'OPS_SUPPORT', 'READERS_PROD'] %}
+        {% do failures.append("Test 2 FAILED: mixed-case normalization expected ['ANALYST', 'OPS_SUPPORT', 'READERS_PROD'], got " ~ result) %}
+    {% endif %}
+
+    {# Test 3: already-uppercase roles stay unchanged #}
+    {% set result = dbt_dataengineers_utils._grants_normalize_roles(['ADMIN', 'DATAOPS_ADMIN']) %}
+    {% if result != ['ADMIN', 'DATAOPS_ADMIN'] %}
+        {% do failures.append("Test 3 FAILED: uppercase passthrough expected ['ADMIN', 'DATAOPS_ADMIN'], got " ~ result) %}
+    {% endif %}
+
+    {# Test 4: empty list returns empty list #}
+    {% set result = dbt_dataengineers_utils._grants_normalize_roles([]) %}
+    {% if result != [] %}
+        {% do failures.append("Test 4 FAILED: empty list expected [], got " ~ result) %}
+    {% endif %}
+
+    {# Test 5: single role #}
+    {% set result = dbt_dataengineers_utils._grants_normalize_roles(['my_Role']) %}
+    {% if result != ['MY_ROLE'] %}
+        {% do failures.append("Test 5 FAILED: single role expected ['MY_ROLE'], got " ~ result) %}
+    {% endif %}
+
+    {# ── Test _grants_format_list ── #}
+
+    {# Test 6: formats a list of values into quoted comma-separated string #}
+    {% set result = dbt_dataengineers_utils._grants_format_list(['SCHEMA_A', 'SCHEMA_B']) %}
+    {% if result != "'SCHEMA_A', 'SCHEMA_B'" %}
+        {% do failures.append("Test 6 FAILED: format_list expected \"'SCHEMA_A', 'SCHEMA_B'\", got " ~ result) %}
+    {% endif %}
+
+    {# Test 7: empty list returns empty string #}
+    {% set result = dbt_dataengineers_utils._grants_format_list([]) %}
+    {% if result != '' %}
+        {% do failures.append("Test 7 FAILED: format_list empty expected '', got " ~ result) %}
+    {% endif %}
+
+    {# Test 8: single item list #}
+    {% set result = dbt_dataengineers_utils._grants_format_list(['PUBLIC']) %}
+    {% if result != "'PUBLIC'" %}
+        {% do failures.append("Test 8 FAILED: format_list single expected \"'PUBLIC'\", got " ~ result) %}
+    {% endif %}
+
+    {# Test 9: escapes single quotes within values #}
+    {% set result = dbt_dataengineers_utils._grants_format_list(["it's"]) %}
+    {% if result != "'it''s'" %}
+        {% do failures.append("Test 9 FAILED: format_list escaping expected \"'it''s'\", got " ~ result) %}
+    {% endif %}
+
+    {# ── Test _grants_append_unique ── #}
+
+    {# Test 10: appends unique value #}
+    {% set test_list = ['A', 'B'] %}
+    {% do dbt_dataengineers_utils._grants_append_unique(test_list, 'C') %}
+    {% if test_list != ['A', 'B', 'C'] %}
+        {% do failures.append("Test 10 FAILED: append_unique expected ['A', 'B', 'C'], got " ~ test_list) %}
+    {% endif %}
+
+    {# Test 11: does not append duplicate #}
+    {% set test_list = ['A', 'B', 'C'] %}
+    {% do dbt_dataengineers_utils._grants_append_unique(test_list, 'B') %}
+    {% if test_list != ['A', 'B', 'C'] %}
+        {% do failures.append("Test 11 FAILED: append_unique duplicate expected ['A', 'B', 'C'], got " ~ test_list) %}
+    {% endif %}
+
+    {# ── Test case-insensitive role comparisons ── #}
+
+    {# Test 12: normalized role list membership check #}
+    {% set roles = dbt_dataengineers_utils._grants_normalize_roles(['analyst', 'Developer']) %}
+    {% if 'ANALYST' not in roles %}
+        {% do failures.append("Test 12 FAILED: 'ANALYST' should be in normalized roles") %}
+    {% endif %}
+    {% if 'DEVELOPER' not in roles %}
+        {% do failures.append("Test 12b FAILED: 'DEVELOPER' should be in normalized roles") %}
+    {% endif %}
+    {% if 'analyst' in roles %}
+        {% do failures.append("Test 12c FAILED: 'analyst' (lowercase) should NOT be in normalized roles") %}
+    {% endif %}
+
+    {# Test 13: grantee_name from Snowflake (uppercase) matches normalized roles #}
+    {% set roles = dbt_dataengineers_utils._grants_normalize_roles(['ops_support']) %}
+    {% set snowflake_grantee = 'OPS_SUPPORT' %}
+    {% if snowflake_grantee not in roles %}
+        {% do failures.append("Test 13 FAILED: Snowflake grantee 'OPS_SUPPORT' should match normalized ['ops_support']") %}
+    {% endif %}
+
+    {# ── Report results ── #}
+    {% if failures | length > 0 %}
+        {% for f in failures %}
+            {% do log(f, info=True) %}
+        {% endfor %}
+        {{ exceptions.raise_compiler_error("test_grants_helpers: " ~ (failures | length) ~ " test(s) failed. See log above.") }}
+    {% else %}
+        {% do log("test_grants_helpers: all 13 tests passed", info=True) %}
+    {% endif %}
+{% endmacro %}

integration_tests/macros/test_grants_idempotency.sql

@@ -0,0 +1,127 @@
+{#
+  Integration test: verifies grant macros are idempotent (skip when no changes needed).
+  Run via: dbt run-operation test_grants_idempotency
+  Requires a Snowflake connection. Uses grants_dry_run=true to avoid mutations.
+  Returns nothing on success; raises an error on any assertion failure.
+
+  This test:
+  1. Runs grant macros once to establish state
+  2. Captures the log output / statement counts
+  3. Verifies that the macros correctly identify "no changes required" scenarios
+#}
+{% macro test_grants_idempotency() %}
+    {% if flags.WHICH not in ['run', 'run-operation'] %}
+        {% do log('test_grants_idempotency: skipped (context)', info=True) %}
+        {% do return(none) %}
+    {% endif %}
+    {% if not execute %}
+        {% do log('test_grants_idempotency: skipped (compile phase)', info=True) %}
+        {% do return(none) %}
+    {% endif %}
+
+    {% set failures = [] %}
+    {% set test_schema = 'PUBLIC' %}
+
+    {# ── Test 1: _grants_get_schema_grants returns a list ── #}
+    {% set roles_with_usage = dbt_dataengineers_utils._grants_get_schema_grants(test_schema, 'USAGE', 'ROLE') %}
+    {% if roles_with_usage is not iterable %}
+        {% do failures.append("Test 1 FAILED: _grants_get_schema_grants should return a list, got " ~ roles_with_usage) %}
+    {% else %}
+        {% do log("Test 1 PASSED: _grants_get_schema_grants returned " ~ (roles_with_usage | length) ~ " roles with USAGE on " ~ test_schema, info=True) %}
+    {% endif %}
+
+    {# ── Test 2: _grants_get_schema_object_privs returns a dict ── #}
+    {% set privs = dbt_dataengineers_utils._grants_get_schema_object_privs(test_schema, ['SELECT'], []) %}
+    {% if privs is not mapping %}
+        {% do failures.append("Test 2 FAILED: _grants_get_schema_object_privs should return a dict, got " ~ privs) %}
+    {% else %}
+        {% do log("Test 2 PASSED: _grants_get_schema_object_privs returned dict with " ~ (privs.keys() | list | length) ~ " roles", info=True) %}
+    {% endif %}
+
+    {# ── Test 3: _grants_get_future_grants returns a dict ── #}
+    {% set future = dbt_dataengineers_utils._grants_get_future_grants(test_schema) %}
+    {% if future is not mapping %}
+        {% do failures.append("Test 3 FAILED: _grants_get_future_grants should return a dict, got " ~ future) %}
+    {% else %}
+        {% do log("Test 3 PASSED: _grants_get_future_grants returned dict with " ~ (future.keys() | list | length) ~ " roles", info=True) %}
+    {% endif %}
+
+    {# ── Test 4: _grants_collect_schemas returns non-empty list ── #}
+    {% set schemas = dbt_dataengineers_utils._grants_collect_schemas(['INFORMATION_SCHEMA'], is_exclude_list=true) %}
+    {% if schemas | length == 0 %}
+        {% do failures.append("Test 4 FAILED: _grants_collect_schemas returned no schemas (database may be empty)") %}
+    {% else %}
+        {% do log("Test 4 PASSED: _grants_collect_schemas found " ~ (schemas | length) ~ " schemas", info=True) %}
+    {% endif %}
+
+    {# ── Test 5: grant_schema_monitor_specific with existing role skips ── #}
+    {# Find a role that already has MONITOR grants in the test schema #}
+    {% set monitor_query %}
+        select distinct grantee
+        from information_schema.object_privileges
+        where privilege_type = 'MONITOR' and object_schema = '{{ test_schema }}'
+        limit 1
+    {% endset %}
+    {% set monitor_results = run_query(monitor_query) %}
+    {% if monitor_results and monitor_results | length > 0 %}
+        {% set existing_role = monitor_results[0][0] %}
+        {% do log("Test 5: Found existing MONITOR role: " ~ existing_role ~ ", running monitor_specific with dry_run", info=True) %}
+        {# Running with the role that already has MONITOR should produce minimal/no new statements #}
+        {{ dbt_dataengineers_utils.grant_schema_monitor_specific([test_schema], [existing_role], false, true) }}
+        {% do log("Test 5 PASSED: grant_schema_monitor_specific completed without error for existing role", info=True) %}
+    {% else %}
+        {% do log("Test 5 SKIPPED: no existing MONITOR grants found in " ~ test_schema, info=True) %}
+    {% endif %}
+
+    {# ── Test 6: grant_schema_read_specific runs without error (dry-run equivalent) ── #}
+    {# Use a role that likely already has SELECT on PUBLIC #}
+    {% set select_query %}
+        select distinct grantee
+        from information_schema.object_privileges
+        where privilege_type = 'SELECT' and object_schema = '{{ test_schema }}'
+        limit 1
+    {% endset %}
+    {% set select_results = run_query(select_query) %}
+    {% if select_results and select_results | length > 0 %}
+        {% set existing_role = select_results[0][0] %}
+        {% do log("Test 6: Running grant_schema_read_specific for role " ~ existing_role ~ " which already has SELECT", info=True) %}
+        {# revoke_current_grants=false prevents any mutations; just tests the skip-logic path #}
+        {{ dbt_dataengineers_utils.grant_schema_read_specific([test_schema], [existing_role], false, false) }}
+        {% do log("Test 6 PASSED: grant_schema_read_specific completed without error", info=True) %}
+    {% else %}
+        {% do log("Test 6 SKIPPED: no existing SELECT grants found in " ~ test_schema, info=True) %}
+    {% endif %}
+
+    {# ── Test 7: Case-insensitive role matching works against live data ── #}
+    {% if select_results and select_results | length > 0 %}
+        {% set existing_role = select_results[0][0] %}
+        {# Pass lowercase version of the role - should still match #}
+        {% set lower_role = existing_role | lower %}
+        {% set normalized = dbt_dataengineers_utils._grants_normalize_roles([lower_role]) %}
+        {% if normalized[0] != existing_role | upper %}
+            {% do failures.append("Test 7 FAILED: normalize_roles('" ~ lower_role ~ "') should be '" ~ (existing_role | upper) ~ "', got " ~ normalized[0]) %}
+        {% else %}
+            {% do log("Test 7 PASSED: normalize_roles('" ~ lower_role ~ "') == '" ~ normalized[0] ~ "'", info=True) %}
+        {% endif %}
+    {% else %}
+        {% do log("Test 7 SKIPPED: no roles to test", info=True) %}
+    {% endif %}
+
+    {# ── Test 8: grant_schema_procedure_usage_specific runs without error ── #}
+    {{ dbt_dataengineers_utils.grant_schema_procedure_usage_specific([test_schema], ['SYSADMIN'], false, true) }}
+    {% do log("Test 8 PASSED: grant_schema_procedure_usage_specific completed without error (dry_run=true)", info=True) %}
+
+    {# ── Test 9: grant_schema_operate_specific runs without error ── #}
+    {{ dbt_dataengineers_utils.grant_schema_operate_specific([test_schema], ['SYSADMIN'], false, true) }}
+    {% do log("Test 9 PASSED: grant_schema_operate_specific completed without error (dry_run=true)", info=True) %}
+
+    {# ── Report results ── #}
+    {% if failures | length > 0 %}
+        {% for f in failures %}
+            {% do log(f, info=True) %}
+        {% endfor %}
+        {{ exceptions.raise_compiler_error("test_grants_idempotency: " ~ (failures | length) ~ " test(s) failed. See log above.") }}
+    {% else %}
+        {% do log("test_grants_idempotency: all tests passed", info=True) %}
+    {% endif %}
+{% endmacro %}