Skip to content

SNOW-2345238: Re-use row count from relaxed query compiler#3791

Merged
sfc-gh-joshi merged 2 commits into
mainfrom
joshi-SNOW-2345238-dedup-count
Sep 17, 2025
Merged

SNOW-2345238: Re-use row count from relaxed query compiler#3791
sfc-gh-joshi merged 2 commits into
mainfrom
joshi-SNOW-2345238-dedup-count

Conversation

@sfc-gh-joshi

Copy link
Copy Markdown
Contributor
  1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.

    Fixes SNOW-2345238

  2. Fill out the following pre-review checklist:

    • I am adding a new automated test(s) to verify correctness of my new code
      • If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing
    • I am adding new logging messages
    • I am adding a new telemetry message
    • I am adding new credentials
    • I am adding a new dependency
    • If this is a new feature/behavior, I'm adding the Local Testing parity changes.
    • I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: Thread-safe Developer Guidelines
    • If adding any arguments to public Snowpark APIs or creating new public Snowpark APIs, I acknowledge that I have ensured my changes include AST support. Follow the link for more information: AST Support Guidelines
  3. Please describe how your code solves the related issue.

SNOW-2230971 (#3687) introduced a second "relaxed" query compiler with placeholder row positions that exists in parallel to the existing Snowflake QC. When the Snowflake QC calls create_initial_ordered_dataframe, it may issue a metadata query to retrieve a row count if the frame is created from a Snowflake query on a table. However, after the addition of the relaxed QC, this code path was hit twice (here), and a second redundant row count query was issued. This PR re-uses the result of the first row count query when possible to do so.

This regression was not caught by testing because the SQL counter explicitly filters out row count metadata queries. Accordingly, this PR makes no testing changes because the row count reduction is not visible to the SQL counter as currently configured. Moreover, the regression was not caught by the benchmarking job (which does NOT filter row count queries) because it stopped running due to crashes caused by enabling hybrid execution on main. I verified the removal of the extra row count query manually, and its impact will be visible in the benchmark dashboard.

The removal of this query has a non-trivial impact on operations with short runtimes, where network latency becomes a meaningful factor. This will address the regression of single-API benchmarks like pd.read_snowflake, which went from ~0.2s to ~0.6s for a 1e6x10 table between commits.

@sfc-gh-joshi sfc-gh-joshi added NO-PANDAS-CHANGEDOC-UPDATES This PR does not update Snowpark pandas docs and removed snowpark-pandas labels Sep 17, 2025

@sfc-gh-mvashishtha sfc-gh-mvashishtha left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

@sfc-gh-joshi sfc-gh-joshi merged commit 20b4d51 into main Sep 17, 2025
34 of 35 checks passed
@sfc-gh-joshi sfc-gh-joshi deleted the joshi-SNOW-2345238-dedup-count branch September 17, 2025 20:23
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 17, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

NO-PANDAS-CHANGEDOC-UPDATES This PR does not update Snowpark pandas docs snowpark-pandas

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants