Skip to content

Commit d54ea01

Browse files
jonhopper-dataengineersCortex Code
jonhopper-dataengineers
and
Cortex Code
authored
Skip grant statements for non-existent object types and already-granted privileges (v1.0.9) (#37)
* Skip grant statements for non-existent object types and already-granted privileges Grant macros now detect which object types exist in each schema before issuing GRANT ON ALL statements. Schemas without a given object type (e.g. no streams, no stages) no longer receive unnecessary grants. Also checks existing privileges per role and skips grants already applied. Affects: grant_schema_read_specific, grant_schema_monitor_specific, grant_schema_operate_specific. Adds _grants_get_schema_object_types helper. Bumped version to 1.0.9. .... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-Authored-By: Cortex Code <noreply@snowflake.com> * Use bulk object type detection to avoid per-schema SHOW queries Replaced per-schema SHOW STREAMS/STAGES/PIPES/TASKS with a single bulk query via _grants_get_all_schema_object_types() that runs just 2 queries total (information_schema.tables + object_privileges). Also replaced individual SHOW commands in _grants_get_schema_object_types with information_schema.object_privileges lookup (2 queries vs 5). .... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-Authored-By: Cortex Code <noreply@snowflake.com> * Count-based coverage check and bulk view lookup for shares Replaced naive privilege-exists check with count-based full coverage check (_grants_get_schema_full_coverage): compares granted object count vs total object count per type. Only re-grants when a new object was added that doesn't yet have the privilege. Replaced per-schema SHOW VIEWS in grant_internal_share_read with a single bulk query against information_schema.tables for all views. .... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-Authored-By: Cortex Code <noreply@snowflake.com> * Update changelog for v1.0.9 with full performance improvements .... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-Authored-By: Cortex Code <noreply@snowflake.com> * Fix coverage query: resolve object_type mismatch for views Snowflake stores views as object_type='TABLE' in object_privileges but information_schema.tables categorizes them as table_type='VIEW'. The coverage query now joins granted_counts with information_schema.tables to resolve the actual object type, ensuring views are correctly matched and their grants are detected as fully covered. .... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-Authored-By: Cortex Code <noreply@snowflake.com> * Bulk coverage check: single query for all schemas instead of per-schema The per-schema _grants_get_schema_full_coverage query took ~14s each, totalling ~9 minutes for 38 schemas. Replaced with bulk _grants_get_all_schema_full_coverage that runs a single database-wide query and returns results keyed by schema. grant_schema_read_specific now runs 3 bulk queries upfront (object types, coverage, roles) then iterates schemas with pure dict lookups. Only SHOW GRANTS ON SCHEMA remains per-schema (~0.3s each). .... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-Authored-By: Cortex Code <noreply@snowflake.com> * Optimize grant_internal_share_read: dict lookups and pre-built schema set Converted existing_share_objects from list (O(n) scans) to dict (O(1) lookups). Pre-built schemas_with_table_grants during DESC SHARE parsing to eliminate the per-schema startswith loop. Added logging for executed statements to aid debugging share grant timing. .... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-Authored-By: Cortex Code <noreply@snowflake.com> * Guard run_query calls with execute check to prevent parse-time failures All helper macros now return early when not in execute phase, preventing run_query from being called during dbt compile/parse/docs. Addresses sourcery-ai review comment on PR #37. .... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-Authored-By: Cortex Code <noreply@snowflake.com> --------- Co-authored-by: Cortex Code <noreply@snowflake.com>
1 parent cdcb72b commit d54ea01

8 files changed

Lines changed: 517 additions & 120 deletions

‎CHANGELOG.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,17 @@
11
# Data Engineers Snowflake DataOps Utils Project Changelog
22
This file contains the changelog for the Data Engineers Snowflake DataOps Utils project, detailing updates, fixes, and enhancements made to the project over time.
33

4+
## v1.0.9 - 2026-06-26 - Conditional Grant Statements & Performance
5+
6+
### Changed
7+
- **Performance**: `grant_schema_read_specific` now detects which object types exist in each schema and only issues grant statements for types that are present. Schemas with no objects of a given type no longer receive unnecessary `GRANT ... ON ALL` statements.
8+
- **Performance**: `grant_schema_read_specific` uses count-based full coverage checking — compares the number of objects with the privilege granted vs total objects of that type. Only re-issues `GRANT ON ALL` when a new object was added that doesn't yet have the privilege. On steady-state runs (no new objects), zero grant statements are executed.
9+
- **Performance**: `grant_schema_monitor_specific` and `grant_schema_operate_specific` now skip schemas that have no pipes or tasks, and only issue grants for the specific object types that exist.
10+
- **Performance**: Object type detection uses bulk database-wide queries (`_grants_get_all_schema_object_types`) — 2 queries total instead of per-schema SHOW commands.
11+
- **Performance**: `grant_internal_share_read` replaced per-schema `SHOW VIEWS` with a single bulk `information_schema.tables` query for all views across the database.
12+
- Added helper macros: `_grants_get_schema_object_types`, `_grants_get_all_schema_object_types`, `_grants_get_schema_full_coverage`.
13+
- Bumped version from 1.0.8 to 1.0.9
14+
415
## v1.0.8 - 2026-06-25 - Grant SQL Quoting Fix
516

617
### Fixed

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
A macro-only [dbt](https://github.com/dbt-labs/dbt) package for Snowflake DataOps. Provides utilities for object lifecycle management, RBAC grant orchestration, dimensional modelling helpers, tagging, shares, and more.
44

5-
- **Version**: 1.0.8
5+
- **Version**: 1.0.9
66
- **dbt**: `>=1.3.0, <3.0.0`
77
- **Dependencies**: None (zero external package dependencies)
88
- **dbt Fusion**: Compatible

‎dbt_project.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
name: 'dbt_dataengineers_utils'
2-
version: '1.0.8'
2+
version: '1.0.9'
33
config-version: 2
44

55
require-dbt-version: [">=1.9.4", "<3.0.0"]

‎macros/grants/_helpers.sql‎

Lines changed: 283 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -156,6 +156,190 @@ These ideas intentionally deferred to keep current refactor incremental.
156156
{% do return(result_map) %}
157157
{% endmacro %}
158158

159+
{# Bulk version: check full privilege coverage across ALL schemas in the database in a single query.
160+
Returns dict {schema: {role: [priv:type_keys_with_full_coverage]}}.
161+
A privilege is "fully covered" when granted_count >= total_count for that object type. #}
162+
{% macro _grants_get_all_schema_full_coverage(privilege_types, grantees) %}
163+
{% set schema_map = {} %}
164+
{% set priv_filter = privilege_types | map('upper') | list %}
165+
{% set grantees_upper = grantees | map('upper') | list %}
166+
{% if not execute or grantees_upper | length == 0 or priv_filter | length == 0 %}
167+
{% do return(schema_map) %}
168+
{% endif %}
169+
{% set query %}
170+
with object_counts as (
171+
select table_schema as schema_name, obj_type, count(*) as total_count
172+
from (
173+
select
174+
table_schema,
175+
case
176+
when table_type = 'VIEW' then 'VIEW'
177+
when table_type = 'MATERIALIZED VIEW' then 'MATERIALIZED VIEW'
178+
when table_type = 'EXTERNAL TABLE' then 'EXTERNAL TABLE'
179+
when is_dynamic = 'YES' then 'DYNAMIC TABLE'
180+
when table_type = 'BASE TABLE' then 'TABLE'
181+
else table_type
182+
end as obj_type,
183+
table_name as object_name
184+
from information_schema.tables
185+
where table_schema != 'INFORMATION_SCHEMA'
186+
union all
187+
select object_schema as table_schema, object_type as obj_type, object_name
188+
from (
189+
select distinct object_schema, object_type, object_name
190+
from information_schema.object_privileges
191+
where object_type in ('STREAM', 'STAGE')
192+
and grantor is not null
193+
)
194+
) all_objects
195+
group by table_schema, obj_type
196+
),
197+
granted_counts as (
198+
select
199+
op.object_schema,
200+
op.grantee,
201+
op.privilege_type,
202+
case
203+
when t.table_type = 'VIEW' then 'VIEW'
204+
when t.table_type = 'MATERIALIZED VIEW' then 'MATERIALIZED VIEW'
205+
when t.table_type = 'EXTERNAL TABLE' then 'EXTERNAL TABLE'
206+
when t.is_dynamic = 'YES' then 'DYNAMIC TABLE'
207+
when t.table_type = 'BASE TABLE' then 'TABLE'
208+
when t.table_type is not null then t.table_type
209+
else op.object_type
210+
end as resolved_type,
211+
count(distinct op.object_name) as granted_count
212+
from information_schema.object_privileges op
213+
left join information_schema.tables t
214+
on op.object_name = t.table_name
215+
and op.object_schema = t.table_schema
216+
where op.privilege_type in ('{{ priv_filter | join("', '") }}')
217+
and op.grantee in ('{{ grantees_upper | join("', '") }}')
218+
and op.grantor is not null
219+
group by op.object_schema, op.grantee, op.privilege_type, resolved_type
220+
)
221+
select
222+
g.object_schema,
223+
g.grantee,
224+
g.privilege_type,
225+
g.resolved_type as object_type
226+
from granted_counts g
227+
inner join object_counts o
228+
on g.object_schema = o.schema_name
229+
and g.resolved_type = o.obj_type
230+
where g.granted_count >= o.total_count
231+
{% endset %}
232+
{% set results = run_query(query) %}
233+
{% if execute and results %}
234+
{% for row in results %}
235+
{% set s = row[0] %}
236+
{% set role = row[1] %}
237+
{% set priv = row[2] %}
238+
{% set obj_type = row[3] %}
239+
{% set key = priv ~ ':' ~ obj_type %}
240+
{% if schema_map.get(s) is none %}
241+
{% set _ = schema_map.update({s: {}}) %}
242+
{% endif %}
243+
{% if schema_map.get(s).get(role) is none %}
244+
{% set _ = schema_map.get(s).update({role: []}) %}
245+
{% endif %}
246+
{% if key not in schema_map.get(s).get(role) %}
247+
{% do schema_map.get(s).get(role).append(key) %}
248+
{% endif %}
249+
{% endfor %}
250+
{% endif %}
251+
{% do return(schema_map) %}
252+
{% endmacro %}
253+
254+
{# Check if roles have FULL privilege coverage on all objects in a schema.
255+
Returns dict {role: [priv_types_with_full_coverage]}.
256+
A privilege type is "fully covered" when the count of objects with that grant equals the total count of objects. #}
257+
{% macro _grants_get_schema_full_coverage(schema, privilege_types, grantees) %}
258+
{% set result_map = {} %}
259+
{% set priv_filter = privilege_types | map('upper') | list %}
260+
{% set grantees_upper = grantees | map('upper') | list %}
261+
{% if not execute or grantees_upper | length == 0 or priv_filter | length == 0 %}
262+
{% do return(result_map) %}
263+
{% endif %}
264+
{% set query %}
265+
with object_counts as (
266+
select obj_type, count(*) as total_count
267+
from (
268+
select
269+
case
270+
when table_type = 'VIEW' then 'VIEW'
271+
when table_type = 'MATERIALIZED VIEW' then 'MATERIALIZED VIEW'
272+
when table_type = 'EXTERNAL TABLE' then 'EXTERNAL TABLE'
273+
when is_dynamic = 'YES' then 'DYNAMIC TABLE'
274+
when table_type = 'BASE TABLE' then 'TABLE'
275+
else table_type
276+
end as obj_type,
277+
table_name as object_name
278+
from information_schema.tables
279+
where table_schema = '{{ schema }}'
280+
union all
281+
select object_type as obj_type, object_name
282+
from (
283+
select distinct object_type, object_name
284+
from information_schema.object_privileges
285+
where object_schema = '{{ schema }}'
286+
and object_type in ('STREAM', 'STAGE')
287+
and grantor is not null
288+
)
289+
) all_objects
290+
group by obj_type
291+
),
292+
granted_counts as (
293+
select
294+
op.grantee,
295+
op.privilege_type,
296+
case
297+
when t.table_type = 'VIEW' then 'VIEW'
298+
when t.table_type = 'MATERIALIZED VIEW' then 'MATERIALIZED VIEW'
299+
when t.table_type = 'EXTERNAL TABLE' then 'EXTERNAL TABLE'
300+
when t.is_dynamic = 'YES' then 'DYNAMIC TABLE'
301+
when t.table_type = 'BASE TABLE' then 'TABLE'
302+
when t.table_type is not null then t.table_type
303+
else op.object_type
304+
end as resolved_type,
305+
count(distinct op.object_name) as granted_count
306+
from information_schema.object_privileges op
307+
left join information_schema.tables t
308+
on op.object_name = t.table_name
309+
and op.object_schema = t.table_schema
310+
where op.object_schema = '{{ schema }}'
311+
and op.privilege_type in ('{{ priv_filter | join("', '") }}')
312+
and op.grantee in ('{{ grantees_upper | join("', '") }}')
313+
and op.grantor is not null
314+
group by op.grantee, op.privilege_type, resolved_type
315+
)
316+
select
317+
g.grantee,
318+
g.privilege_type,
319+
g.resolved_type as object_type
320+
from granted_counts g
321+
inner join object_counts o
322+
on g.resolved_type = o.obj_type
323+
where g.granted_count >= o.total_count
324+
{% endset %}
325+
{% set results = run_query(query) %}
326+
{% if execute and results %}
327+
{% for row in results %}
328+
{% set role = row[0] %}
329+
{% set priv = row[1] %}
330+
{% set obj_type = row[2] %}
331+
{% set key = priv ~ ':' ~ obj_type %}
332+
{% if result_map.get(role) is none %}
333+
{% set _ = result_map.update({role: []}) %}
334+
{% endif %}
335+
{% if key not in result_map.get(role) %}
336+
{% do result_map.get(role).append(key) %}
337+
{% endif %}
338+
{% endfor %}
339+
{% endif %}
340+
{% do return(result_map) %}
341+
{% endmacro %}
342+
159343
{# Check schema-level grants (USAGE etc) for given roles. Returns list of roles (uppercased) that already have the privilege. #}
160344
{% macro _grants_get_schema_grants(schema, privilege, grantee_type) %}
161345
{% set existing = [] %}
@@ -198,6 +382,105 @@ These ideas intentionally deferred to keep current refactor incremental.
198382
{% do return(result_map) %}
199383
{% endmacro %}
200384

385+
{# Detect which object types exist in a schema. Returns a list of type strings.
386+
Types returned: 'TABLE', 'VIEW', 'MATERIALIZED VIEW', 'EXTERNAL TABLE', 'DYNAMIC TABLE', 'STREAM', 'STAGE', 'PIPE', 'TASK' #}
387+
{% macro _grants_get_schema_object_types(schema) %}
388+
{% set object_types = [] %}
389+
{% if not execute %}
390+
{% do return(object_types) %}
391+
{% endif %}
392+
{# Table-like objects from information_schema.tables #}
393+
{% set query %}
394+
select distinct
395+
case
396+
when is_dynamic = 'YES' then 'DYNAMIC TABLE'
397+
when table_type = 'BASE TABLE' then 'TABLE'
398+
when table_type = 'EXTERNAL TABLE' then 'EXTERNAL TABLE'
399+
when table_type = 'MATERIALIZED VIEW' then 'MATERIALIZED VIEW'
400+
else table_type
401+
end as object_type
402+
from information_schema.tables
403+
where table_schema = '{{ schema }}'
404+
{% endset %}
405+
{% set results = run_query(query) %}
406+
{% if results %}
407+
{% for row in results %}
408+
{% if row[0] not in object_types %}
409+
{% do object_types.append(row[0]) %}
410+
{% endif %}
411+
{% endfor %}
412+
{% endif %}
413+
{# Non-table objects: query information_schema.object_privileges for existence (avoids SHOW commands) #}
414+
{% set other_types_query %}
415+
select distinct object_type
416+
from information_schema.object_privileges
417+
where object_schema = '{{ schema }}'
418+
and object_type in ('STREAM', 'STAGE', 'PIPE', 'TASK')
419+
{% endset %}
420+
{% set other_results = run_query(other_types_query) %}
421+
{% if other_results %}
422+
{% for row in other_results %}
423+
{% if row[0] not in object_types %}
424+
{% do object_types.append(row[0]) %}
425+
{% endif %}
426+
{% endfor %}
427+
{% endif %}
428+
{% do return(object_types) %}
429+
{% endmacro %}
430+
431+
{# Bulk version: detect object types for ALL schemas in the database in two queries.
432+
Returns a dict {schema_name: [object_types]}. Call once per run and pass the result around. #}
433+
{% macro _grants_get_all_schema_object_types() %}
434+
{% set schema_map = {} %}
435+
{% if not execute %}
436+
{% do return(schema_map) %}
437+
{% endif %}
438+
{# Table-like objects #}
439+
{% set query %}
440+
select table_schema,
441+
case
442+
when is_dynamic = 'YES' then 'DYNAMIC TABLE'
443+
when table_type = 'BASE TABLE' then 'TABLE'
444+
when table_type = 'EXTERNAL TABLE' then 'EXTERNAL TABLE'
445+
when table_type = 'MATERIALIZED VIEW' then 'MATERIALIZED VIEW'
446+
else table_type
447+
end as object_type
448+
from information_schema.tables
449+
group by table_schema, object_type
450+
{% endset %}
451+
{% set results = run_query(query) %}
452+
{% if execute and results %}
453+
{% for row in results %}
454+
{% set s = row[0] %}
455+
{% if schema_map.get(s) is none %}
456+
{% set _ = schema_map.update({s: []}) %}
457+
{% endif %}
458+
{% if row[1] not in schema_map.get(s) %}
459+
{% do schema_map.get(s).append(row[1]) %}
460+
{% endif %}
461+
{% endfor %}
462+
{% endif %}
463+
{# Non-table objects from object_privileges #}
464+
{% set other_query %}
465+
select distinct object_schema, object_type
466+
from information_schema.object_privileges
467+
where object_type in ('STREAM', 'STAGE', 'PIPE', 'TASK')
468+
{% endset %}
469+
{% set other_results = run_query(other_query) %}
470+
{% if execute and other_results %}
471+
{% for row in other_results %}
472+
{% set s = row[0] %}
473+
{% if schema_map.get(s) is none %}
474+
{% set _ = schema_map.update({s: []}) %}
475+
{% endif %}
476+
{% if row[1] not in schema_map.get(s) %}
477+
{% do schema_map.get(s).append(row[1]) %}
478+
{% endif %}
479+
{% endfor %}
480+
{% endif %}
481+
{% do return(schema_map) %}
482+
{% endmacro %}
483+
201484
{# Row formatters for specific ownership object types #}
202485
{% macro _fmt_schema_ownership(row) %}
203486
grant ownership on schema {{ target.database }}.{{ row[0] }} to role {{ var('current_role_name_override', '') or '' }} revoke current grants;

0 commit comments

Comments
 (0)