SNOW-2220946: Add support for unstructured data engineering in Snowpark by sfc-gh-jdu · Pull Request #3775 · snowflakedb/snowpark-python

sfc-gh-jdu · 2025-09-12T17:14:51Z

Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.

Fixes SNOW-2220946
Fill out the following pre-review checklist:
- I am adding a new automated test(s) to verify correctness of my new code
  - If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing
- I am adding new logging messages
- I am adding a new telemetry message
- I am adding new credentials
- I am adding a new dependency
- If this is a new feature/behavior, I'm adding the Local Testing parity changes.
- I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: Thread-safe Developer Guidelines
- If adding any arguments to public Snowpark APIs or creating new public Snowpark APIs, I acknowledge that I have ensured my changes include AST support. Follow the link for more information: AST Support Guidelines
Please describe how your code solves the related issue.

see changelog

…_agg` (#3679)

…timent` and `DataFrame.ai.summarize_agg` (#3705)

…se_document` (#3710)

sfc-gh-jkew · 2025-09-12T17:25:31Z

-    WriteParquet write_parquet = 182;
-    WriteSave write_save = 183;
-    WriteTable write_table = 184;
+    DataframeAiAgg dataframe_ai_agg = 47;


Are we concerned with these constants changing? Normally I would add to the end and increment to prevent version incompatibilities but I'm not sure if it matters with how the AST is used today.

AST change was already reviewed in this PR https://github.com/snowflakedb/snowpark-python/pull/3761/files

This file is autogenerated, the constants shouldn't pose any issues until we start versioning the AST.

sfc-gh-aalam · 2025-09-15T16:50:30Z

@@ -1,5 +1,29 @@
 # Release History

+## 1.40.0 (YYYY-MM-DD)


you don't want this to go out with 1.39.0?

Yea 1.39 will be this week, right? I'm not sure whether we can finish review by this week.

sfc-gh-yuwang · 2025-09-16T17:26:55Z

+    def __init__(self, dataframe: "snowflake.snowpark.DataFrame") -> None:
+        self._dataframe = dataframe
+
+    @experimental(version="1.40.0")


this also need to be changed to 1.39 if want to included in this release

sfc-gh-yuwang · 2025-09-16T17:27:10Z

+            df._ast_id = stmt.uid
+        return df
+
+    @experimental(version="1.40.0")


sfc-gh-yuwang · 2025-09-16T17:27:20Z

+            filtered_df._ast_id = stmt.uid
+        return filtered_df
+
+    @experimental(version="1.40.0")


sfc-gh-yuwang · 2025-09-16T17:27:30Z

+            df._ast_id = stmt.uid
+        return df
+
+    @experimental(version="1.40.0")


sfc-gh-yuwang · 2025-09-16T17:27:37Z

+            df._ast_id = stmt.uid
+        return df
+
+    @experimental(version="1.40.0")


sfc-gh-yuwang · 2025-09-16T17:27:44Z

+            df._ast_id = stmt.uid
+        return df
+
+    @experimental(version="1.40.0")


sfc-gh-yuwang · 2025-09-16T17:27:50Z

+            df._ast_id = stmt.uid
+        return df
+
+    @experimental(version="1.40.0")


sfc-gh-yuwang · 2025-09-16T17:28:16Z

+            df._ast_id = stmt.uid
+        return df
+
+    @experimental(version="1.40.0")


sfc-gh-yuwang · 2025-09-16T17:28:24Z

+            df._ast_id = stmt.uid
+        return df
+
+    @experimental(version="1.40.0")


sfc-gh-yuwang · 2025-09-16T17:28:50Z

+            df._ast_id = stmt.uid
+        return df
+
+    @experimental(version="1.40.0")


There are also some place need to change below

sfc-gh-heshah

LGTM

sfc-gh-jdu added 11 commits September 11, 2025 17:21

SNOW-2254812: Add DataFrame.ai.complete (basic functionality) (#3651)

d0ba587

SNOW-2262408: DataFrame.ai.filter (#3672)

e9da473

SNOW-2270322: Support DataFrame.ai.agg and `DataFrame.group_by().ai…

16853e0

…_agg` (#3679)

SNOW-2270328: Support DataFrame.ai.classify (#3690)

ca3bf31

SNOW-2270330: DataFrame.ai.similarity (#3692)

1457f38

SNOW-2270334: Add support for DataFrame.ai.embed, `DataFrame.ai.sen…

561a8e8

…timent` and `DataFrame.ai.summarize_agg` (#3705)

SNOW-2301877: Support DataFrame.ai.transcribe and `DataFrame.ai.par…

89f320d

…se_document` (#3710)

SNOW-2301878: Support DataFrame.ai.extract (#3719)

4ff162d

SNOW-2306960: Add three helper functions in DataFrame.ai (#3732)

74e4ccb

SNOW-2263127: Add AST support for AI functions (#3761)

3c3da83

Add changelog and bump experimental version of ai functions (#3774)

15f3a5f

sfc-gh-jdu requested review from a team as code owners September 12, 2025 17:14

sfc-gh-jdu requested review from sfc-gh-mayliu, sfc-gh-yixie and sfc-gh-yuwang September 12, 2025 17:14

sfc-gh-jkew reviewed Sep 12, 2025

View reviewed changes

update test

acd154b

sfc-gh-aalam reviewed Sep 15, 2025

View reviewed changes

sfc-gh-aalam approved these changes Sep 15, 2025

View reviewed changes

sfc-gh-yuwang reviewed Sep 16, 2025

View reviewed changes

sfc-gh-yuwang approved these changes Sep 16, 2025

View reviewed changes

version

c10d064

sfc-gh-heshah approved these changes Sep 16, 2025

View reviewed changes

sfc-gh-jdu merged commit b77dc73 into main Sep 16, 2025
26 of 29 checks passed

sfc-gh-jdu deleted the ai branch September 16, 2025 20:03

github-actions Bot locked and limited conversation to collaborators Sep 16, 2025

Conversation

sfc-gh-jdu commented Sep 12, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sfc-gh-heshah left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants