Skip to content

Reduce Snowflake profiler output size by decoupling query metrics from QUERY_TEXT#2567

Merged
dgomez04 merged 3 commits into
mainfrom
fix/snowflake-profiler-query-text-bloat
Jul 17, 2026
Merged

Reduce Snowflake profiler output size by decoupling query metrics from QUERY_TEXT#2567
dgomez04 merged 3 commits into
mainfrom
fix/snowflake-profiler-query-text-bloat

Conversation

@dgomez04

@dgomez04 dgomez04 commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Summary

This pull request addresses the massive Snowflake profiler extract sizes on high-volume accounts by decoupling metrics tracking from raw SQL text query history. We preserve high-value metrics while drastically reducing the bloat.

Key Changes

  • Omit QUERY_TEXT from full-window query_history. Prevents multi-gigabyte SQL text extraction while keeping the exact QUERY_TYPE and performance metrics.
  • Introduce query_samples. Extracts a random sample of up to 10,000 rows containing QUERY_TEXT and QUERY_ID (to join back to query_history for downstream migration complexity analysis).
  • Drop unused user_activity. Removed the LOGIN_HISTORYdump, which is no longer used by lakebridge_profilers or GVP inputs and was contributing millions of rows of secondary bloat.

Measured Impact

On a sample ~20M-row extract, omitting the raw SQL text yielded dramatic storage and network transfer savings:

Metric Before After Change
Extract Size ~28 GiB ~1 GiB ~96% reduction (28× smaller)

Note. Workload spend-mix classification relies on QUERY_TYPE, not QUERY_TEXT, meaning downstream classification accuracy remains unaffected.

Follow-Ups

  • Updated lakebridge-profilers' migration_complexity.sql to read from query_samples instead of query_history.QUERY_TEXT.

Tests

  • Execute execute-database-profiler on Snowflake staging/test accounts and verified:
    - [ ] query_history no longer contains query_text
    - [ ] query_samples is limited to $\le$ 10k rows and contains the text.
    - [ ] user_activity table is completely gone.
  • Confirm that lakebridge-profilers (after changes) spend-mix path still functions correctly using QUERY_TYPE from query_history.

Closes #2532

Keep exact QUERY_TYPE metrics in query_history, add a 10k-row query_samples
table with SQL text for downstream heuristics, and drop unused user_activity
login dumps that inflate high-volume extracts (#2532).
@dgomez04
dgomez04 requested a review from a team as a code owner July 14, 2026 16:35
@codecov

codecov Bot commented Jul 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 70.35%. Comparing base (c1fdb1f) to head (734847c).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2567   +/-   ##
=======================================
  Coverage   70.35%   70.35%           
=======================================
  Files         107      107           
  Lines        9658     9658           
  Branches     1069     1069           
=======================================
  Hits         6795     6795           
  Misses       2655     2655           
  Partials      208      208           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@dgomez04
dgomez04 requested review from jneil17 and m-abulazm July 14, 2026 16:39
@dgomez04 dgomez04 changed the title Bound Snowflake profiler extract size (#2532) Bound Snowflake Profiler Extract Size Jul 14, 2026
@dgomez04 dgomez04 changed the title Bound Snowflake Profiler Extract Size Reduce Snowflake profiler output size by decoupling query metrics from QUERY_TEXT Jul 14, 2026
@github-actions

github-actions Bot commented Jul 14, 2026

Copy link
Copy Markdown

✅ 174/174 passed, 2 skipped, 1h40m13s total

Running from acceptance #5084

@dgomez04 dgomez04 self-assigned this Jul 14, 2026
@dgomez04 dgomez04 added the feat/profiler Issues related to profilers label Jul 14, 2026
@dgomez04
dgomez04 enabled auto-merge July 16, 2026 14:46

@gueniai gueniai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dgomez04
dgomez04 added this pull request to the merge queue Jul 17, 2026
Merged via the queue into main with commit 8ba21c1 Jul 17, 2026
15 checks passed
@dgomez04
dgomez04 deleted the fix/snowflake-profiler-query-text-bloat branch July 17, 2026 17:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feat/profiler Issues related to profilers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[ISSUE]: Snowflake profiler extract is too large on high-volume accounts

3 participants