Skip to content

Commit 0c7e0ff

Browse files
robert3005claude
robert3005
and
claude
authored
Add docs on how to use Spark vortex datasource (#8728)
We have refactored how the spark datasource is laid out and some of the instructions were incorrect. At the same time we were not handling ALL of the spark usage patterns that others have grown accustomed to Co-authored-by: Claude <noreply@anthropic.com>
1 parent 862f158 commit 0c7e0ff

9 files changed

Lines changed: 737 additions & 15 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
[![CodSpeed Badge](https://img.shields.io/endpoint?url=https://codspeed.io/badge.json)](https://codspeed.io/vortex-data/vortex)
77
[![Crates.io](https://img.shields.io/crates/v/vortex.svg)](https://crates.io/crates/vortex)
88
[![PyPI - Version](https://img.shields.io/pypi/v/vortex-data)](https://pypi.org/project/vortex-data/)
9-
[![Maven - Version](https://img.shields.io/maven-central/v/dev.vortex/vortex-spark)](https://central.sonatype.com/artifact/dev.vortex/vortex-spark)
9+
[![Maven - Version](https://img.shields.io/maven-central/v/dev.vortex/vortex-spark_2.13)](https://central.sonatype.com/artifact/dev.vortex/vortex-spark_2.13)
1010
[![codecov](https://codecov.io/github/vortex-data/vortex/graph/badge.svg)](https://codecov.io/github/vortex-data/vortex)
1111
[![Cite](https://img.shields.io/badge/cite-CITATION.cff-blue)](CITATION.cff)
1212

‎docs/user-guide/spark.md‎

Lines changed: 113 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,34 +1,71 @@
11
# Spark
22

33
Vortex provides a Spark DataSource V2 connector for reading and writing Vortex files. The
4-
connector is published to Maven Central as `dev.vortex:vortex-spark`.
4+
connector is published to Maven Central in two flavors:
55

6-
## Installation
6+
- `dev.vortex:vortex-spark_2.13` for Spark 4.x (Scala 2.13)
7+
- `dev.vortex:vortex-spark_2.12` for Spark 3.5.x (Scala 2.12)
78

8-
Add the dependency to your build. The connector is built against Spark 4.x with Scala 2.13.
9+
Use the `all` classifier JAR (e.g. `vortex-spark_2.13-0.78.0-all.jar`). It is self-contained:
10+
it bundles the Vortex JNI bindings, native libraries for Linux (x86_64 and aarch64) and macOS
11+
(aarch64), and relocates its Arrow, Guava, and Jackson dependencies to avoid classpath
12+
conflicts with Spark. The thin (unclassified) JAR does not work on its own because it
13+
references relocated classes that only ship in the `all` JAR.
14+
15+
## Getting Vortex into Spark
16+
17+
For `spark-shell`, `spark-submit`, or `pyspark`, pass the `all` JAR with `--jars`. Spark
18+
accepts either a local path or a URL, so you can point directly at Maven Central:
19+
20+
```shell
21+
spark-shell --jars https://repo1.maven.org/maven2/dev/vortex/vortex-spark_2.13/0.78.0/vortex-spark_2.13-0.78.0-all.jar
22+
```
23+
24+
Or equivalently when building a session programmatically, e.g. in PySpark:
25+
26+
```python
27+
spark = (
28+
SparkSession.builder
29+
.config("spark.jars", "/path/to/vortex-spark_2.13-0.78.0-all.jar")
30+
.getOrCreate()
31+
)
32+
```
33+
34+
```{note}
35+
`--packages dev.vortex:vortex-spark_2.13:0.78.0` does not work: `--packages` cannot select
36+
the `all` classifier and resolves the thin JAR, which fails at runtime with
37+
`NoClassDefFoundError: dev/vortex/relocated/...`.
38+
```
39+
40+
Once the JAR is on the classpath, the connector registers itself automatically under the
41+
format name `vortex` — no session configuration is required.
42+
43+
## Installation as a Build Dependency
44+
45+
To depend on the connector from a JVM project, add the `all` classifier to the dependency:
946

1047
````{tab} Gradle (Kotlin)
1148
```kotlin
12-
implementation("dev.vortex:vortex-spark:<version>")
49+
implementation("dev.vortex:vortex-spark_2.13:0.78.0:all")
1350
```
1451
````
1552

1653
````{tab} Maven
1754
```xml
1855
<dependency>
1956
<groupId>dev.vortex</groupId>
20-
<artifactId>vortex-spark</artifactId>
21-
<version>${vortex.version}</version>
57+
<artifactId>vortex-spark_2.13</artifactId>
58+
<version>0.78.0</version>
59+
<classifier>all</classifier>
2260
</dependency>
2361
```
2462
````
2563

26-
The connector ships as a shadow JAR that relocates its Arrow, Guava, and Protobuf dependencies
27-
to avoid classpath conflicts with Spark.
28-
2964
## Reading Vortex Files
3065

31-
Use the `vortex` format to read a single file or a directory of Vortex files:
66+
Paths may be local filesystem paths (`/path/to/data`) or URLs (`file:///path/to/data`,
67+
`s3://bucket/path/to/data`). Use the `vortex` format to read a single file or a directory of
68+
Vortex files:
3269

3370
```java
3471
Dataset<Row> df = spark.read()
@@ -65,6 +102,72 @@ Each Spark partition produces one output file named `part-{partitionId}-{taskId}
65102
The connector supports all standard Spark save modes: `Overwrite`, `Append`, `Ignore`, and
66103
`ErrorIfExists`.
67104

105+
## Spark SQL
106+
107+
The connector can also be used from pure SQL. To query existing Vortex files, register them
108+
as a temporary view:
109+
110+
```sql
111+
CREATE TEMPORARY VIEW people
112+
USING vortex
113+
OPTIONS (path '/path/to/data');
114+
115+
SELECT name, age FROM people WHERE age > 30;
116+
```
117+
118+
Tables can be created with `USING vortex`, then written to and read back with plain SQL.
119+
With a `LOCATION` clause the table is external, backed by the files at that path; without
120+
one the table is managed, and Spark stores its data under the warehouse directory (and
121+
deletes it on `DROP TABLE`):
122+
123+
```sql
124+
CREATE TABLE student (id INT, name STRING, age INT)
125+
USING vortex;
126+
127+
INSERT INTO student VALUES (1, 'Alice', 20), (2, 'Bob', 21);
128+
129+
SELECT * FROM student;
130+
```
131+
132+
`CREATE TABLE ... AS SELECT` works the same way:
133+
134+
```sql
135+
CREATE TABLE adults
136+
USING vortex
137+
AS SELECT * FROM people WHERE age >= 18;
138+
```
139+
140+
```{note}
141+
On Spark 3.5, `CREATE TABLE ... USING vortex` additionally requires replacing the session
142+
catalog, because Spark 3.5's built-in catalog cannot read tables backed by a DataSource
143+
V2-only connector:
144+
145+
spark.sql.catalog.spark_catalog=dev.vortex.spark.VortexSessionCatalog
146+
147+
The extension delegates everything to the built-in session catalog (including the Hive
148+
metastore, if configured) and only changes how `vortex` tables are resolved; tables of other
149+
providers are untouched. It is not needed on Spark 4, though setting it is harmless.
150+
```
151+
152+
## Direct File Queries
153+
154+
Spark's built-in ``SELECT * FROM format.`path` `` syntax only works for built-in file
155+
formats, so the connector ships a path-based catalog that provides the equivalent for
156+
Vortex. Register it in the session configuration under the name `vortex`:
157+
158+
```shell
159+
spark-sql --conf spark.sql.catalog.vortex=dev.vortex.spark.VortexCatalog
160+
```
161+
162+
Then query a Vortex file, or a directory of Vortex files, directly by path — no view or
163+
table required:
164+
165+
```sql
166+
SELECT * FROM vortex.`/path/to/data`;
167+
168+
INSERT INTO vortex.`/path/to/data` VALUES (1, 'Alice', 20);
169+
```
170+
68171
## Supported Types
69172

70173
| Spark Type | Vortex Type |

‎java/README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,9 @@
33
We provide two interfaces for working with Vortex from Java:
44

55
- `vortex-java` - a low-level interface JNI for working with Vortex files and arrays on cloud and local storage
6-
- `vortex-spark` - A Spark connector for working with datasets of Vortex files
6+
- `vortex-spark` - A Spark connector for working with datasets of Vortex files. See
7+
[vortex-spark/README.md](vortex-spark/README.md) for how to load the connector into Spark
8+
and query Vortex files from the DataFrame API or Spark SQL.
79

810
## Publishing
911

‎java/vortex-spark/README.md‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
# vortex-spark
2+
3+
A Spark DataSource V2 connector for reading and writing [Vortex](https://vortex.dev) files.
4+
It registers itself under the format name `vortex` and supports both the DataFrame API and
5+
Spark SQL.
6+
7+
Two flavors are published to Maven Central:
8+
9+
| Artifact | Spark | Scala |
10+
|-------------------------------|-----------|-------|
11+
| `dev.vortex:vortex-spark_2.13` | Spark 4.x | 2.13 |
12+
| `dev.vortex:vortex-spark_2.12` | Spark 3.5.x | 2.12 |
13+
14+
Use the `all` classifier JAR (e.g. `vortex-spark_2.13-0.78.0-all.jar`). It is self-contained:
15+
it bundles the Vortex JNI bindings, native libraries for Linux (x86_64 and aarch64) and macOS
16+
(aarch64), and relocates its Arrow, Guava, and Jackson dependencies to avoid classpath
17+
conflicts with Spark. The thin (unclassified) JAR does not work on its own because it
18+
references relocated classes that only ship in the `all` JAR.
19+
20+
## Getting Vortex into Spark
21+
22+
Pass the `all` JAR to `spark-shell`, `spark-submit`, or `pyspark` with `--jars`. Spark accepts
23+
either a local path or a URL, so you can point directly at Maven Central:
24+
25+
```shell
26+
spark-shell --jars https://repo1.maven.org/maven2/dev/vortex/vortex-spark_2.13/0.78.0/vortex-spark_2.13-0.78.0-all.jar
27+
```
28+
29+
Or configure it on the session builder, e.g. in PySpark:
30+
31+
```python
32+
spark = (
33+
SparkSession.builder
34+
.config("spark.jars", "/path/to/vortex-spark_2.13-0.78.0-all.jar")
35+
.getOrCreate()
36+
)
37+
```
38+
39+
Note that `--packages dev.vortex:vortex-spark_2.13:0.78.0` does not work: `--packages` cannot
40+
select the `all` classifier and resolves the thin JAR, which fails at runtime with
41+
`NoClassDefFoundError: dev/vortex/relocated/...`.
42+
43+
To depend on the connector from a JVM project instead, add the `all` classifier to the
44+
dependency:
45+
46+
```kotlin
47+
implementation("dev.vortex:vortex-spark_2.13:0.78.0:all")
48+
```
49+
50+
## Usage
51+
52+
Paths may be local filesystem paths (`/path/to/data`) or URLs (`file:///path/to/data`,
53+
`s3://bucket/path/to/data`).
54+
55+
### DataFrame API
56+
57+
```java
58+
// Write
59+
df.write()
60+
.format("vortex")
61+
.option("path", "/path/to/output")
62+
.mode(SaveMode.Overwrite)
63+
.save();
64+
65+
// Read a single file or a directory of .vortex files
66+
Dataset<Row> df = spark.read()
67+
.format("vortex")
68+
.option("path", "/path/to/output")
69+
.load();
70+
```
71+
72+
### Spark SQL
73+
74+
```sql
75+
-- Query existing Vortex files through a temporary view
76+
CREATE TEMPORARY VIEW people
77+
USING vortex
78+
OPTIONS (path '/path/to/data');
79+
80+
SELECT name, age FROM people WHERE age > 30;
81+
82+
-- Create a table and write to it. With a LOCATION clause the table is external,
83+
-- backed by the files at that path; without one it is managed by Spark.
84+
CREATE TABLE student (id INT, name STRING, age INT)
85+
USING vortex;
86+
87+
INSERT INTO student VALUES (1, 'Alice', 20), (2, 'Bob', 21);
88+
89+
SELECT * FROM student;
90+
```
91+
92+
On Spark 3.5, `CREATE TABLE ... USING vortex` additionally requires replacing the session
93+
catalog with `spark.sql.catalog.spark_catalog=dev.vortex.spark.VortexSessionCatalog`,
94+
because Spark 3.5's built-in catalog cannot read tables backed by a DataSource V2-only
95+
connector. The extension delegates everything else to the built-in session catalog and
96+
leaves tables of other providers untouched; it is not needed on Spark 4.
97+
98+
### Direct file queries
99+
100+
Spark's built-in ``SELECT * FROM format.`path` `` syntax only works for built-in file
101+
formats, so the connector ships a path-based catalog that provides the equivalent for
102+
Vortex. Register it under the name `vortex` with this session config:
103+
104+
```
105+
spark.sql.catalog.vortex=dev.vortex.spark.VortexCatalog
106+
```
107+
108+
Then query (or insert into) Vortex files directly by path:
109+
110+
```sql
111+
SELECT * FROM vortex.`/path/to/data`;
112+
```
113+
114+
See the [Spark user guide](https://docs.vortex.dev/user-guide/spark.html) for the full
115+
documentation, including supported types, write options, and S3 configuration.

0 commit comments

Comments
 (0)