Skip to content

NO-SNOW - Update the min python version for modin precommit to 3.10#3800

Closed
sfc-gh-jkew wants to merge 2660 commits into
mainfrom
jkew/modin_pre_commit_python_update
Closed

NO-SNOW - Update the min python version for modin precommit to 3.10#3800
sfc-gh-jkew wants to merge 2660 commits into
mainfrom
jkew/modin_pre_commit_python_update

Conversation

@sfc-gh-jkew

@sfc-gh-jkew sfc-gh-jkew commented Sep 23, 2025

Copy link
Copy Markdown
Contributor

  1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.

    Fixes SNOW-NNNNNNN

  2. Fill out the following pre-review checklist:

    • I am adding a new automated test(s) to verify correctness of my new code
      • If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing
    • I am adding new logging messages
    • I am adding a new telemetry message
    • I am adding new credentials
    • I am adding a new dependency
    • If this is a new feature/behavior, I'm adding the Local Testing parity changes.
    • I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: Thread-safe Developer Guidelines
    • If adding any arguments to public Snowpark APIs or creating new public Snowpark APIs, I acknowledge that I have ensured my changes include AST support. Follow the link for more information: AST Support Guidelines

  3. Please describe how your code solves the related issue.

    Please write a short description of how your code change solves the related issue.

sfc-gh-helmeleegy and others added 30 commits June 13, 2025 13:52
@sfc-gh-helmeleegy
SNOW-2104775: Improve error messages for unsupported modin or pandas …
908a63e 
…versions when used in Notebooks (#3413)
@sfc-gh-joshi
NO-SNOW: Fix dependency setup issues in modin daily tests (#3467)
57bed56
@sfc-gh-daviwang
SNOW-2096130 : Adding sql query line tracking to SnowflakePlan and S…
2776ae5 
…electable (#3416)
@sfc-gh-aalam
SNOW-2150430: Flatten session.table.sample (#3466)
50b33a8
@sfc-gh-helmeleegy
SNOW-2135473, SNOW-2149908: Add support for DataFrame/Series.to_excel (…
20d02de 
…#3465)
@sfc-gh-daviwang
SNOW-2096094: Maintaining list of ast_ids for SnowflakePlan (#3472)
fc918e0
@sfc-gh-jkew @sfc-gh-mvashishtha
SNOW-1865516 - Add modin api telemetry to snowpark pandas (#3468)
b0c3fff 
Co-authored-by: Mahesh Vashishtha <mahesh.vashishtha@snowflake.com>
@sfc-gh-yuwang
SNOW-2157867:add mongodb to test resource (#3475)
64ec768
@sfc-gh-jkew @sfc-gh-helmeleegy
SNOW-2046347: Add hybrid execution documentation as a top-level topic (…
4169d5e 
…#3474)

Co-authored-by: Hazem Elmeleegy <hazem.elmeleegy@snowflake.com>
@sfc-gh-jrose @graphite-app
SNOW-2097586: Add Debug mode for eager evaluation (#3380)
8d1f99c 
Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
@sfc-gh-daviwang
SNOW-2108634: Adding python source trace to debug context (#3469)
e325a51
@sfc-gh-jrose
SNOW-2097591: Add validation to metadata inference (#3392)
bd1dbb2
@sfc-gh-jrose
SNOW-2150552: Add TRY_CAST and USE_RELAXED_TYPES options to DataFrame…
ca87172 
…Reader (#3473)
@sfc-gh-yuwang @sfc-gh-helmeleegy
@sfc-gh-jdu @sfc-gh-joshi @sfc-gh-jrose @sfc-gh-daviwang @sfc-gh-yixie
NO SNOW: Merge 1.33.0 changelog back to main (#3481)
0208903 
Co-authored-by: Hazem Elmeleegy <hazem.elmeleegy@snowflake.com>
Co-authored-by: Jianzhun Du <68252326+sfc-gh-jdu@users.noreply.github.com>
Co-authored-by: Jonathan Shi <149419494+sfc-gh-joshi@users.noreply.github.com>
Co-authored-by: Jamison <jamison.rose@snowflake.com>
Co-authored-by: David Wang <da.wang@snowflake.com>
Co-authored-by: Yijun Xie <yijun.xie@snowflake.com>
@sfc-gh-jkew @graphite-app
SNOW-2130723 - Set enforce_ordering=True when using DML/DDL changes (#…
7c8e88d 
…3484)

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
@sfc-gh-yuwang
SNOW-2173700: update connection parameter to migrate from preprod12 t…
fa30935 
…o preprod6 (#3486)
@sfc-gh-yuwang
NO SNOW: update recipe (#3485)
336e2f0
@sfc-gh-aalam
SNOW-2094325: Added fix for explode to support complex inputs (#3480)
e7675cb
@sfc-gh-yuwang
SNOW-2160042: remove redundant validation in iceberg table (#3482)
f3be14b
@sfc-gh-jkew
SNOW-2157605 - Test Only - Providing more specific coverage for a cus…
0826a68 
…tomer-reported issue (#3483)
@sfc-gh-jrose @graphite-app
SNOW-1473621: Add support for mapInPandas (#3476)
b14b276 
Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
@sfc-gh-jkew @sfc-gh-mvashishtha
SNOW-2157718 - fillna attempts to fill incompatible columns with a va…
849999d 
…lue (#3477)

Co-authored-by: Mahesh Vashishtha <mahesh.vashishtha@snowflake.com>
@sfc-gh-aling
SNOW-2174093: improve error handling when closing cursor and connecti…
ad697b7 
…on in dbapi (#3487)
@sfc-gh-joshi
SNOW-2173644, SNOW-2157873: Fix __array_function__ hybrid switching a…
ee257b0 
…nd native Series constructor switching bugs (#3498)

SNOW-2157873 occurs because upstream modin does not implement __array_function__, instead converting to numpy ndarrays via __array__ when a numpy function is called on it. The presence of the extension wrapper for __array_function__ introduced by Snowpark pandas confuses numpy dispatch, causing unexpected AttributeErrors. This is fixed upstream with modin-project/modin#7617, and will presumably become available in the next modin release. On the Snowpark side, this PR adds relevant tests, and adds a version-guarded flag to remove the extension function and push it down to the query compiler.

SNOW-2173644 occurs in specific circumstances when determining switching conditions for the DataFrame constructor. Series objects are treated as dict-like, but Series.values is a property rather than a function. We thus skip over native_pd.Series objects in the dict-like check in move_to_me_cost.
@sfc-gh-lju
[SNOW-2110572]: implement snowflakeFile methods for local paths (#3464)
6a115bf
@sfc-gh-aling
NO-SNOW: fix temp table txn (#3502)
da23473
@sfc-gh-aling
SNOW-2097818: use multithread as the default implementation for dbapi (…
8476f0b 
…#3491)
@sfc-gh-jkew
SNOW-2022983 - Add dtype checks to the move_to_me_cost function for S…
8e96bc3 
…nowpandas (#3496)
@sfc-gh-aalam
SNOW-2113721: Fix overlapping columns with table function (#3499)
c0c846a
@sfc-gh-jkew
SNOW-2177453 - Fix issue with df.melt when there are duplicate intern…
261ba61 
…al column names (#3501)
sfc-gh-joshi and others added 27 commits September 11, 2025 14:33
@sfc-gh-joshi
SNOW-2331021: Wrap internal Series/DF creation (#3768)
895131f
@sfc-gh-aherreraaguilar
SNOW-2298586: Add support for context functions (part 1) (#3730)
3862f97
@sfc-gh-lmukhopadhyay
SNOW-2044877: fix snowpandas in apply error msg (#3678)
1ee82d7 
Signed-off-by: Labanya Mukhopadhyay <labanya.mukhopadhyay@snowflake.com>
@sfc-gh-aling
SNOW-2331878: auto release branch preparation (#3764)
1e8c32e
@sfc-gh-aling
SNOW-2333389: deprecate warning for 39 (#3776)
27cb7d1
@sfc-gh-helmeleegy
SNOW-2325421: Add session parameter dummy_row_pos_optimization_enabled (
57b3dd2 
#3779)
@sfc-gh-yuwang
SNOW-2241912: add test coverage of dbapi on stored procedure (#3739)
cad209b
@sfc-gh-joshi
SNOW-2333472: More hybrid benchmark fixes (#3771)
15c3ae5
@sfc-gh-aalam
SNOW-2211956 support copy grants with create replace view (#3755)
2388e37
@sfc-gh-jdu
SNOW-2220946: Add support for unstructured data engineering in Snowpa…
f972575 
…rk (#3775)
@sfc-gh-fhe
SNOW-2296280: Add interval_day_time_from_parts function (#3702)
050e313
@sfc-gh-joshi
SNOW-2345238: Re-use row count from relaxed query compiler (#3791)
87f6e8f 
SNOW-2230971 (#3687) introduced a second "relaxed" query compiler with placeholder row positions that exists in parallel to the existing Snowflake QC. When the Snowflake QC calls create_initial_ordered_dataframe, it may issue a metadata query to retrieve a row count if the frame is created from a Snowflake query on a table. However, after the addition of the relaxed QC, this code path was hit twice (here), and a second redundant row count query was issued. This PR re-uses the result of the first row count query when possible to do so.

This regression was not caught by testing because the SQL counter explicitly filters out row count metadata queries. Accordingly, this PR makes no testing changes because the row count reduction is not visible to the SQL counter as currently configured. Moreover, the regression was not caught by the benchmarking job (which does NOT filter row count queries) because it stopped running due to crashes caused by enabling hybrid execution on main. I verified the removal of the extra row count query manually, and its impact will be visible in the benchmark dashboard.

The removal of this query has a non-trivial impact on operations with short runtimes, where network latency becomes a meaningful factor. This will address the regression of single-API benchmarks like pd.read_snowflake, which went from ~0.2s to ~0.6s for a 1e6x10 table between commits.
@sfc-gh-yuwang
NO SNOW: fix daily precommit to include localtesting (#3790)
b89f38a
@sfc-gh-yuwang
NO SNOW: merge 1.39.0 back to main (#3792)
b586c3b
@sfc-gh-mvashishtha
SNOW-2331792: Support DataFrame.eval for single-level index. (#3782)
917577d 
Support all the parameters of `DataFrame.eval` via the Modin implementation. Only support single-level row index for now because the MultiIndex is currently a pandas object, so we have to get resolvers for its levels differently.

Signed-off-by: sfc-gh-mvashishtha <mahesh.vashishtha@snowflake.com>
@sfc-gh-mayliu
Snow 2324796 secrets api migration 1 (#3781)
4bce735
@sfc-gh-helmeleegy
SNOW-2338542: Add session parameter pandas_hybrid_execution_enabled (#…
4aa4ab3 
…3789)
@sfc-gh-aherreraaguilar
SNOW-2335536: Add support for Geospatial functions - part 2 (#3786)
6e5b18c
@sfc-gh-yuwang
SNOW-2303618: fix DataFrame.limit() fail if there is parameter bind…
5d9e398 
…ing in the executed SQL. (#3780)
@sfc-gh-aherreraaguilar
SNOW-2334682: Add support for scalar functions from different categor…
0e65150 
…ies. (#3773)
@sfc-gh-aherreraaguilar @graphite-app
SNOW-2314365: Added support for conditional expression functions (#3767)
c08d937 
Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
@sfc-gh-aling
SNOW-2336332: faster release automate fast path release package valid…
63cbb8b 
…ation (#3788)
@sfc-gh-aherreraaguilar @graphite-app
SNOW-2333111: Add support for Geospatial functions - part 1 (#3769)
6b25e5c 
Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
@sfc-gh-jrose
NO-SNOW: Fix local test on windows. (#3796)
4a8faba
@sfc-gh-mvashishtha @sfc-gh-joshi @sfc-gh-helmeleegy
SNOW-800922: Support DataFrame.query for single-level index. (#3795)
4cb3687 
Support all the parameters of `DataFrame.query` via the Modin implementation. Only support single-level row index for now because the MultiIndex is currently a pandas object, so we have to get resolvers for its levels differently.

Signed-off-by: sfc-gh-mvashishtha <mahesh.vashishtha@snowflake.com>
Co-authored-by: Jonathan Shi <149419494+sfc-gh-joshi@users.noreply.github.com>
Co-authored-by: Hazem Elmeleegy <hazem.elmeleegy@snowflake.com>
@sfc-gh-yuwang
SNOW-2313015: add support for session init statement in udtf ingestio…
b92cec5 
…n of dbapi (#3748)
@sfc-gh-jkew
Update the min python version for modin precommit to 3.10
7316a0e
@sfc-gh-aalam sfc-gh-aalam force-pushed the jkew/modin_pre_commit_python_update branch from 394ff87 to 7316a0e Compare November 12, 2025 23:20
@github-actions github-actions Bot locked and limited conversation to collaborators Nov 12, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.