Skip to content

SNOW-2304052: hybrid execution tests on pre-commit#3778

Closed
sfc-gh-jkew wants to merge 2827 commits into
mainfrom
jkew/add-hybrid-precheck
Closed

SNOW-2304052: hybrid execution tests on pre-commit#3778
sfc-gh-jkew wants to merge 2827 commits into
mainfrom
jkew/add-hybrid-precheck

Conversation

@sfc-gh-jkew

@sfc-gh-jkew sfc-gh-jkew commented Sep 12, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

Adds pre-commit tests for hybrid using a subset of known, fast ( < 0.25 s ) tests. This will help prevent regressions; but it represents around 26k tests normally written for push-down mode. These only execute on ubuntu-aws.

Some tests have been explicitly skipped with the @pytest.mark.skip_hybrid marker because they fail on CI but pass on local testing.

Fixes SNOW-2304052

  • I am adding a new automated test(s) to verify correctness of my new code
    • If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing
  • I am adding new logging messages
  • I am adding a new telemetry message
  • I am adding new credentials
  • I am adding a new dependency
  • If this is a new feature/behavior, I'm adding the Local Testing parity changes.
  • I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: Thread-safe Developer Guidelines
  • If adding any arguments to public Snowpark APIs or creating new public Snowpark APIs, I acknowledge that I have ensured my changes include AST support. Follow the link for more information: AST Support Guidelines

sfc-gh-vilee and others added 30 commits August 25, 2025 09:37
@sfc-gh-vilee
[SNOW-2020306] Update application tracking for notebooks (#3695)
b6e0817
@sfc-gh-kgadomski
SNOW-1943633 - fix displaying timestamps for _show_string_spark, in…
75d623d 
…troduce `_spark_session_tz` param (#3659)
@sfc-gh-yuwang
SNOW-2275424: fixudtf_ingestion does not work in stored proc because …
7f1bd1c 
…function name is out of spec (#3691)
@sfc-gh-aling
NO-SNOW: fix doc test (#3707)
52a3216
@sfc-gh-mayliu
SNOW-1791191: time travel support v2 (#3674)
2a9dc3d
@sfc-gh-jdu
SNOW-2296406: Support ai_extract, ai_transcribe and `ai_parse_doc…
86ced20 
…ument` in functions (#3697)
@sfc-gh-aling
SNOW-2223084: update cloudpickle to v3.1.1 (#3584)
bf7928f
@sfc-gh-jkew
SNOW-2213898 - Update assertion code to convert hybrid Pandas df (#3704)
5cdb0b1
@sfc-gh-jkew @graphite-app
SNOW-2230553 - Reduce Telemetry Overhead when running w/ a Pandas Eng…
873c124 
…ine (#3610)

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
@sfc-gh-joshi
SNOW-2292908: Skip flaky modin hybrid tests on Windows (#3698)
3e9337a
@sfc-gh-aherreraaguilar
SNOW-2262972: Refactor function.py file (#3655)
3d0158f
@sfc-gh-jkew
SNOW-2213898 - Update assertion code to handle test_attrs assertions (#…
37a384c 
…3706)

This is working towards running most of our snowpandas tests with hybrid mode.
@sfc-gh-helmeleegy
SNOW-2296598: Estimate row count only when Hybrid is enabled (#3708)
8627b9e
@sfc-gh-aling
SNOW-641374: support directory (#3709)
dbd6805
@sfc-gh-aalam
NO-SNOW: Pin pytest-rerunfailures due to be release (#3720)
95c2bba
@sfc-gh-aherreraaguilar
SNOW-2249728: Add support for scalar functions (batch 1) (#3714)
8d6db91
@sfc-gh-mayliu
SNOW-2292338 AST support for time travel (#3716)
11cafe0
@sfc-gh-yuwang
SNOW-2166197: add support for jdbc (#3591)
2af770b
@sfc-gh-aling
SNOW-2268238: support copy files (#3712)
cb2fd64
@sfc-gh-aling
NO-SNOW: fix no pandas test failure (#3727)
da411b8
@sfc-gh-jkew
SNOW-2304052 - Create a test parameter to optionally enable hybrid te…
3d92421 
…sts for the integration module (#3715)

The new test parameter is called '--enable_modin_hybrid_mode' which is only applied to the integ
modin module. This is not used yet; but it allows for enabling hybrid in an adhoc way. Eventually there
will be a new pre-commit test which enables hybrid just for the integration modin module.

This change also disables the sql_counter when running under hybrid mode; because virtually no sql queries are issued.
@sfc-gh-mayliu
AST on TimestampTimeZone type casting (#3729)
278a28c
@sfc-gh-yuwang
NO SNOW: fix test error because of naming (#3723)
c5f1a7c
@sfc-gh-aling
SNOW-2301201: use server cursor to fetch data (#3726)
29e796e
@sfc-gh-aherreraaguilar
SNOW-2298578: Add support for scalar functions (Bitwise expression fu…
42f4745 
…nctions) (#3721)
@sfc-gh-aalam
NO-SNOW: Fix tests (#3731)
b425f9d
@sfc-gh-yuwang
NO SNOW: align behavior of session_init_statement in dbapi and jdbc (#…
0857605 
…3734)
@sfc-gh-joshi
NO-SNOW: Actually skip hybrid tests on Windows (#3737)
a1a4bb2
@sfc-gh-aling
NO-SNOW: add missing doc update (#3738)
6f7b9d4
@sfc-gh-jkew
SNOW-2305345 - Eliminate duplicate casing parameter checks in snowpan…
960ba1d 
…das (#3717)

SNOW-2305345 - Eliminate duplicate casing parameter checks in snowpandas

While working on SHOW OBJECT usage to see if we can fetch row size quickly I noticed we issue SHOW PARAMETERS LIKE 'QUOTED_IDENTIFIERS_IGNORE_CASE' IN SESSION queries every time we fetch the session. This is done to issue a warning, but we really only need to do this once.
sfc-gh-jenzhang and others added 27 commits October 22, 2025 16:35
@sfc-gh-jenzhang
SNOW-2359402 Enabled autoswitching on some unsupported args (#3906)
1f195fa
@sfc-gh-jkew
SNOW-2390752 - Series replace fails with KeyError ( Test Only ) (#3929)
80543a5
@sfc-gh-jkew
Merge branch 'main' into jkew/add-hybrid-precheck
ce919fe
@sfc-gh-mvashishtha
SNOW-2454948: Revert 2 modin commits to fix CI. (#3945)
a18ac6b 
This commit reverts 2 commits that broke tests, 1f195fa and 8a24385. 

1f195fa broke several tests. 8a24385 broke tests/integ/modin/hybrid/test_switch_operations.py::test_query_count_no_switch[True] and tests/integ/modin/hybrid/test_switch_operations.py::test_query_count_no_switch[False].

GitHub allowed us to merge the commits because I think I never saved the branch protection rule requiring modin-ubuntu-AWS tests to pass before merging to main. I'm struggling with the branch protection rules so I am still trying to fix them.
@sfc-gh-yuwang
SNOW-2230705: add support for UDF profiler (#3909)
2fb40ce
@sfc-gh-mayliu
SNOW-2335605: support SCLS SPCS secrets usage (#3853)
54fe821
@sfc-gh-helmeleegy
NO-SNOW: Refactor test_np_unique into a class (#3943)
4c6ca78
@sfc-gh-helmeleegy
SNOW-2396047: Support np.percentile(DataFrame | Series) (#3940)
6cea9ac
@sfc-gh-helmeleegy
SNOW-2455993: Added support for concat in faster pandas (#3949)
538b34f
@sfc-gh-mkeller
SNOW-2267482 replacing SnowAPI calls with SQL calls (#3942)
3d55db1
@sfc-gh-aling
NO-SNOW: merge back release v1.41.0 (#3952)
ad6c77a
@sfc-gh-mkeller
remove catalog types and snowflake.core dependency (#3950)
c06a7c9
@sfc-gh-mvashishtha @sfc-gh-joshi @graphite-app
SNOW-2396205: Support random_state in sample. (#3918)
7ca06e1 
Implement the random_state parameter DataFrame.sample and Series.sample by sorting with a seeded random order and then selecting the top `n` or `frac * len(dataset)` rows. We use this solution because we can't use the built-in SAMPLE with SEED for this use case.

Signed-off-by: sfc-gh-mvashishtha <mahesh.vashishtha@snowflake.com>
Co-authored-by: Jonathan Shi <149419494+sfc-gh-joshi@users.noreply.github.com>
Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
@sfc-gh-yuwang
SNOW-2364943: support export external telemetry to event table (#3813)
78f8e2f
@sfc-gh-helmeleegy
SNOW-2432713: Ensure that dummy_row_pos_mode passed to OrderedDataFra…
0efc99c 
…me.join() matches SnowflakeQueryCompiler._dummy_row_pos_mode (#3901)
@sfc-gh-yuwang
NO SNOW: fix external telemetry doc string (#3954)
b97543a
@sfc-gh-helmeleegy
SNOW-2458028: Add support for copy in faster pandas (#3956)
ba84ec7
@sfc-gh-jkew
Merge branch 'main' into jkew/add-hybrid-precheck
0d3f05b
@sfc-gh-helmeleegy
SNOW-2458510: Make faster pandas opt-in instead of opt-out (#3957)
daff82c
@sfc-gh-helmeleegy
SNOW-2464676: Add support for str.isdigit/islower/isupper/istitle/low…
5688126 
…er/upper/title in faster pandas (#3961)
@sfc-gh-mayliu
SNOW-2268220: support ORDER BY ALL (#3912)
2a47b17
@sfc-gh-jkew
Merge branch 'main' into jkew/add-hybrid-precheck
7ff5621
@sfc-gh-helmeleegy
SNOW-2444072: Add support for groupby.apply in faster pandas (2nd att…
b692d71 
…empt) (#3962)
@sfc-gh-jkew
Merge branch 'main' into jkew/add-hybrid-precheck
6db0d87
@sfc-gh-aling
NO-SNOW: remove manual trigger creating test branch (#3965)
ee0d538
@sfc-gh-jenzhang
SNOW-2359402 Enabled autoswitching on some unsupported args (#3953)
67b74d2
@sfc-gh-jkew
Merge branch 'main' into jkew/add-hybrid-precheck
54c617f
@sfc-gh-aalam sfc-gh-aalam force-pushed the jkew/add-hybrid-precheck branch from 1b761a0 to 54c617f Compare November 12, 2025 23:20
@github-actions github-actions Bot locked and limited conversation to collaborators Nov 12, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@sfc-gh-nkumar sfc-gh-nkumar Awaiting requested review from sfc-gh-nkumar sfc-gh-nkumar was automatically assigned from snowflakedb/snowpark-pandas-api

@sfc-gh-lmukhopadhyay sfc-gh-lmukhopadhyay Awaiting requested review from sfc-gh-lmukhopadhyay sfc-gh-lmukhopadhyay was automatically assigned from snowflakedb/snowpark-pandas-api

@sfc-gh-yixie sfc-gh-yixie Awaiting requested review from sfc-gh-yixie sfc-gh-yixie was automatically assigned from snowflakedb/snowpark-python-api-reviewers

@sfc-gh-yuwang sfc-gh-yuwang Awaiting requested review from sfc-gh-yuwang sfc-gh-yuwang was automatically assigned from snowflakedb/snowpark-python-api-reviewers

@sfc-gh-mayliu sfc-gh-mayliu Awaiting requested review from sfc-gh-mayliu sfc-gh-mayliu was automatically assigned from snowflakedb/snowpark-python-api-reviewers

Assignees

No one assigned

Labels

NO-CHANGELOG-UPDATES This pull request does not need to update CHANGELOG.md snowpark-pandas

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.