Skip to content

SNOW-2220946: Add support for unstructured data engineering in Snowpark #3775

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sfc-gh-jdu merged 13 commits into from
Sep 16, 2025
Merged

SNOW-2220946: Add support for unstructured data engineering in Snowpark #3775

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
d0ba587
SNOW-2254812: Add `DataFrame.ai.complete` (basic functionality) (#3651)
sfc-gh-jdu Aug 15, 2025
e9da473
SNOW-2262408: `DataFrame.ai.filter` (#3672)
sfc-gh-jdu Aug 19, 2025
16853e0
SNOW-2270322: Support `DataFrame.ai.agg` and `DataFrame.group_by().ai…
sfc-gh-jdu Aug 20, 2025
ca3bf31
SNOW-2270328: Support `DataFrame.ai.classify` (#3690)
sfc-gh-jdu Aug 21, 2025
1457f38
SNOW-2270330: `DataFrame.ai.similarity` (#3692)
sfc-gh-jdu Aug 22, 2025
561a8e8
SNOW-2270334: Add support for `DataFrame.ai.embed`, `DataFrame.ai.sen…
sfc-gh-jdu Aug 27, 2025
89f320d
SNOW-2301877: Support `DataFrame.ai.transcribe` and `DataFrame.ai.par…
sfc-gh-jdu Aug 28, 2025
4ff162d
SNOW-2301878: Support DataFrame.ai.extract (#3719)
sfc-gh-jdu Sep 2, 2025
74e4ccb
SNOW-2306960: Add three helper functions in `DataFrame.ai` (#3732)
sfc-gh-jdu Sep 3, 2025
3c3da83
SNOW-2263127: Add AST support for AI functions (#3761)
sfc-gh-jdu Sep 12, 2025
15f3a5f
Add changelog and bump experimental version of ai functions (#3774)
sfc-gh-jdu Sep 12, 2025
acd154b
update test
sfc-gh-jdu Sep 12, 2025
c10d064
version
sfc-gh-jdu Sep 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,34 @@
# Release History

## 1.40.0 (YYYY-MM-DD)

@sfc-gh-aalam sfc-gh-aalam Sep 15, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you don't want this to go out with 1.39.0?

@sfc-gh-jdu sfc-gh-jdu Sep 15, 2025

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea 1.39 will be this week, right? I'm not sure whether we can finish review by this week.


### Snowpark Python API Updates

#### New Features

## 1.39.0 (YYYY-MM-DD)

### Snowpark Python API Updates

#### New Features

- Added support for unstructured data engineering in Snowpark, powered by Snowflake AISQL and Cortex functions:
- `DataFrame.ai.complete`: Generate per-row LLM completions from prompts built over columns and files.
- `DataFrame.ai.filter`: Keep rows where an AI classifier returns TRUE for the given predicate.
- `DataFrame.ai.agg`: Reduce a text column into one result using a natural-language task description.
- `RelationalGroupedDataFrame.ai_agg`: Perform the same natural-language aggregation per group.
- `DataFrame.ai.classify`: Assign single or multiple labels from given categories to text or images.
- `DataFrame.ai.similarity`: Compute cosine-based similarity scores between two columns via embeddings.
- `DataFrame.ai.sentiment`: Extract overall and aspect-level sentiment from text into JSON.
- `DataFrame.ai.embed`: Generate VECTOR embeddings for text or images using configurable models.
- `DataFrame.ai.summarize_agg`: Aggregate and produce a single comprehensive summary over many rows.
- `DataFrame.ai.transcribe`: Transcribe audio files to text with optional timestamps and speaker labels.
- `DataFrame.ai.parse_document`: OCR/layout-parse documents or images into structured JSON.
- `DataFrame.ai.extract`: Pull structured fields from text or files using a response schema.
- `DataFrame.ai.count_tokens`: Estimate token usage for a given model and input text per row.
- `DataFrame.ai.split_text_markdown_header`: Split Markdown into hierarchical header-aware chunks.
- `DataFrame.ai.split_text_recursive_character`: Split text into size-bounded chunks using recursive separators.
- `DataFrameReader.file`: Create a DataFrame containing all files from a stage as FILE data type for downstream unstructured data processing.
- Added a new datatype `YearMonthIntervalType` that allows users to create intervals for datetime operations.
- Added a new function `interval_year_month_from_parts` that allows users to easily create `YearMonthIntervalType` without using SQL.
- Added a new datatype `DayTimeIntervalType` that allows users to create intervals for datetime operations.
Expand Down
16 changes: 16 additions & 0 deletions docs/source/snowpark/dataframe.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ DataFrame
DataFrameNaFunctions
DataFrameStatFunctions
DataFrameAnalyticsFunctions
DataFrameAIFunctions

.. rubric:: Methods

Expand Down Expand Up @@ -120,6 +121,20 @@ DataFrame
DataFrameAnalyticsFunctions.compute_lag
DataFrameAnalyticsFunctions.compute_lead
DataFrameAnalyticsFunctions.time_series_agg
DataFrameAIFunctions.agg
DataFrameAIFunctions.classify
DataFrameAIFunctions.complete
DataFrameAIFunctions.count_tokens
DataFrameAIFunctions.embed
DataFrameAIFunctions.extract
DataFrameAIFunctions.filter
DataFrameAIFunctions.parse_document
DataFrameAIFunctions.sentiment
DataFrameAIFunctions.similarity
DataFrameAIFunctions.split_text_markdown_header
DataFrameAIFunctions.split_text_recursive_character
DataFrameAIFunctions.summarize_agg
DataFrameAIFunctions.transcribe
dataframe.map
dataframe.map_in_pandas

Expand All @@ -133,6 +148,7 @@ DataFrame
.. autosummary::
:toctree: api/

DataFrame.ai
DataFrame.columns
DataFrame.na
DataFrame.queries
Expand Down
1 change: 1 addition & 0 deletions docs/source/snowpark/grouping.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ Grouping
:toctree: api/

RelationalGroupedDataFrame.agg
RelationalGroupedDataFrame.ai_agg
RelationalGroupedDataFrame.apply_in_pandas
RelationalGroupedDataFrame.applyInPandas
RelationalGroupedDataFrame.avg
Expand Down
2 changes: 2 additions & 0 deletions src/snowflake/snowpark/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
"DataFrameStatFunctions",
"DataFrameAnalyticsFunctions",
"DataFrameNaFunctions",
"DataFrameAIFunctions",
"DataFrameWriter",
"DataFrameReader",
"GroupingSets",
Expand Down Expand Up @@ -54,6 +55,7 @@
from snowflake.snowpark.column import CaseExpr, Column
from snowflake.snowpark.stored_procedure_profiler import StoredProcedureProfiler
from snowflake.snowpark.dataframe import DataFrame
from snowflake.snowpark.dataframe_ai_functions import DataFrameAIFunctions
from snowflake.snowpark.dataframe_analytics_functions import DataFrameAnalyticsFunctions
from snowflake.snowpark.dataframe_na_functions import DataFrameNaFunctions
from snowflake.snowpark.dataframe_reader import DataFrameReader
Expand Down
Loading
Loading