Skip to content

Commit 2887d32

Browse files
IshiiharaIshiihara
committed
Add support for Snowflake Container Service
1 parent e358195 commit 2887d32

File tree

12 files changed

+229
-9
lines changed

12 files changed

+229
-9
lines changed

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
target/
2+
.idea/

‎Dockerfile‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
FROM openjdk:8-jdk-alpine
2+
WORKDIR /app
3+
COPY bin /app/bin
4+
COPY config /app/config
5+
COPY importlibs /app/importlibs
6+
COPY jdbc /app/jdbc
7+
COPY lib /app/lib
8+
COPY mainPath /app/mainPath
9+
COPY target /app/target
10+
# Set executable permission for esprocx.sh
11+
RUN chmod +x /app/bin/ServerConsole2.sh
12+
ENTRYPOINT ["/bin/sh", "/app/bin/ServerConsole2.sh", "-h"]

‎SNOWFLAKE_README.md‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
## Setting Up the Environment for Snowflake SPL Service
2+
3+
This guide will walk you through installing the required tools, building and uploading the project image, and creating the compute pool and SPL service in Snowflake.
4+
5+
### 1. Install JDK 1.8
6+
7+
The project requires **JDK 1.8**. Please download it from [Oracle's website](https://www.oracle.com/java/technologies/javase/javase-jdk8-downloads.html) and set the appropriate `JAVA_HOME` environment variable.
8+
9+
To verify that JDK is installed correctly, run:
10+
```bash
11+
java -version
12+
```
13+
Make sure the output shows the correct version (Java 1.8).
14+
15+
### 2. Install Maven
16+
17+
You also need to install **Maven**. On macOS, it can be easily installed via [Homebrew](https://brew.sh/):
18+
```bash
19+
brew install maven
20+
```
21+
To verify Maven installation, run:
22+
```bash
23+
mvn -version
24+
```
25+
Ensure that the version output matches your requirements.
26+
27+
### 3. Build and Upload the Docker Image
28+
29+
We provide a script to **build** the project and **upload** the resulting Docker image to a repository managed by Snowflake. You can customize the repository location if needed.
30+
31+
Before running the script, set your Snowflake account as an environment variable:
32+
```bash
33+
export SNOWFLAKE_ACCOUNT=your-snowflake-account
34+
```
35+
Then run the build script:
36+
```bash
37+
./build.sh
38+
```
39+
This script will build the Docker image and upload it to the Snowflake-managed Docker repository.
40+
41+
### 4. Create the Compute Pool and SPL Service in Snowflake
42+
43+
To create the SPL service in Snowflake, run the following SQL command in your Snowflake workbook. This command will create a service that provides endpoints to execute SPL files saved in the specified path.
44+
```sql
45+
CREATE SERVICE spl_service
46+
IN COMPUTE POOL tutorial_compute_pool
47+
FROM SPECIFICATION $$
48+
spec:
49+
container:
50+
- name: main
51+
image: /tutorial_db/data_schema/tutorial_repository/spl_service:latest
52+
env:
53+
SNOWFLAKE_WAREHOUSE: tutorial_warehouse
54+
volumeMounts:
55+
- name: data
56+
mountPath: /opt/data
57+
endpoints:
58+
- name: spl
59+
port: 8502
60+
public: true
61+
volumes:
62+
- name: data
63+
source: "@tutorial_stage"
64+
$$
65+
MIN_INSTANCES=1
66+
MAX_INSTANCES=1;
67+
```
68+
69+
The service will provide endpoints to allow users to execute SPL files located in the main path, which is backed by the stage. As long as the SPL file is in the stage, it can be executed via REST APIs.
70+
71+
To **check the status** of the service or **retrieve logs**, use the following SQL commands:
72+
```sql
73+
SELECT SYSTEM$GET_SERVICE_STATUS('spl_service');
74+
SELECT SYSTEM$GET_SERVICE_LOGS('spl_service', '0', 'main');
75+
```
76+
77+
### 5. Execute SPL Using REST API
78+
79+
To execute SPL via REST API, first get the service endpoint:
80+
```sql
81+
SHOW ENDPOINTS IN SERVICE spl_service;
82+
```
83+
84+
From the result, copy the `ingress_url` and paste it into your web browser (e.g., Chrome). You can then invoke the SPL using a REST API request like this:
85+
```bash
86+
https://your-ingress-url.snowflakecomputing.app/file_test.splx()
87+
```
88+
Replace `your-ingress-url` with the actual URL obtained from the `SHOW ENDPOINTS` command.
89+
90+
This assumes that both the `file_test.splx` and any relevant data are in the `mainPath` backed by the stage.
91+
92+
### 6. Upload Data and SPL Files to Snowflake Stage
93+
94+
To upload data or SPL files into the Snowflake stage, use **SnowSQL** (see [SnowSQL documentation](https://docs.snowflake.com/en/user-guide/snowsql)). For example, to upload the `employee.btx` file:
95+
```sql
96+
PUT file:////path-to-esproc/mainPath/employee.btx @tutorial_stage
97+
AUTO_COMPRESS=FALSE
98+
OVERWRITE=TRUE;
99+
```
100+
Change `path-to-esproc` to the actual path on your machine. This command will place the file into the `tutorial_stage`, making it accessible for SPL execution.
101+
102+
### Summary
103+
104+
In this guide, you've set up your environment, built the project Docker image, created a Snowflake SPL service, and learned how to execute SPL files. Make sure to verify each step, ensuring everything is correctly configured for smooth operation.

‎bin/ServerConsole2.sh‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
#!/bin/bash
2+
source /app/bin/setEnv2.sh
3+
"$EXEC_JAVA" -Xms128m -Xmx1024m -cp "$START_HOME"/app/target/classes:"$START_HOME"/app/lib/*:"$START_HOME"/jdbc/* -Duser.language="$language" -Dstart.home="$START_HOME"/app com.scudata.ide.spl.ServerConsole $1 $2

‎bin/setEnv2.sh‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
START_HOME=/
2+
JAVA_HOME=/usr/bin/
3+
EXEC_JAVA=$JAVA_HOME/java
4+
language=en

‎build.sh‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
#!/bin/bash
2+
3+
# Input arguments with default fallbacks
4+
SNOWFLAKE_ACCOUNT=${1:-$SNOWFLAKE_ACCOUNT}
5+
REPOSITORY_NAME=${2:-$REPOSITORY_NAME}
6+
7+
# Validate inputs
8+
if [ -z "$SNOWFLAKE_ACCOUNT" ]; then
9+
echo "Error: Snowflake account not specified."
10+
echo "Usage: $0 <snowflake_account> [repository_name] or set SNOWFLAKE_ACCOUNT environment variable."
11+
exit 1
12+
fi
13+
14+
if [ -z "$REPOSITORY_NAME" ]; then
15+
REPOSITORY_NAME="tutorial_db/data_schema/tutorial_repository"
16+
echo "Repository not specified. Using default: $REPOSITORY_NAME"
17+
fi
18+
19+
# Construct dynamic variables
20+
IMAGE_HOST="$SNOWFLAKE_ACCOUNT.registry.snowflakecomputing.com"
21+
REPOSITORY="$IMAGE_HOST/$REPOSITORY_NAME"
22+
IMAGE_NAME="$REPOSITORY/spl-service:latest"
23+
24+
# Maven clean and package
25+
echo "Building Maven project..."
26+
mvn clean package -Prelease || { echo "Maven build failed! Exiting."; exit 1; }
27+
28+
# Docker build and push
29+
echo "Building Docker image..."
30+
docker build --rm --platform linux/amd64 -t "$IMAGE_NAME" . || { echo "Docker build failed! Exiting."; exit 1; }
31+
32+
echo "Pushing Docker image..."
33+
docker push "$IMAGE_NAME" || { echo "Docker push failed! Exiting."; exit 1; }
34+
35+
echo "Docker image pushed successfully: $IMAGE_NAME"

‎config/raqsoftConfig.xml‎

Lines changed: 11 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
1-
<?xml version="1.0" encoding="UTF-8"?>
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
23
<Config Version="3">
34
<Runtime>
4-
<DBList>
5+
<DBList>
56
</DBList>
67
<Esproc>
78
<charSet>GBK</charSet>
@@ -11,7 +12,7 @@
1112
<dateFormat>yyyy-MM-dd</dateFormat>
1213
<timeFormat>HH:mm:ss</timeFormat>
1314
<dateTimeFormat>yyyy-MM-dd HH:mm:ss</dateTimeFormat>
14-
<mainPath />
15+
<mainPath>/opt/data</mainPath>>
1516
<tempPath />
1617
<bufSize>65536</bufSize>
1718
<parallelNum>1</parallelNum>
@@ -20,14 +21,15 @@
2021
<nullStrings>nan,null,n/a</nullStrings>
2122
<fetchCount>9999</fetchCount>
2223
<extLibsPath/>
23-
<customFunctionFile>customFunctions.properties</customFunctionFile>
24+
<customFunctionFile>customFunctions.properties</customFunctionFile>
25+
2426
</Esproc>
2527
<Logger>
2628
<Level>INFO</Level>
2729
</Logger>
28-
</Runtime>
29-
<JDBC>
30-
<load>Runtime,Server</load>
31-
<gateway></gateway>
30+
</Runtime>
31+
<JDBC>
32+
<load>Runtime,Server</load>
33+
<gateway></gateway>
3234
</JDBC>
33-
</Config>
35+
</Config>

‎mainPath/demo.splx‎

2.5 KB
Binary file not shown.

‎mainPath/employee.btx‎

240 Bytes
Binary file not shown.

‎mainPath/file_test.splx‎

2.5 KB
Binary file not shown.

0 commit comments

Comments
 (0)