spark

Connecting Java and Apache Spark with ADBC

Instructions

Tip

If you already have a Spark instance running, skip the steps to set up and clean up Spark.

Prerequisites

1. Install Maven

2. Install dbc

Set up Spark

1. Install Docker

2. Start a Spark instance:

   docker run -d --rm --name spark-connect -p 15002:15002 apache/spark:4.1.2 bash -c "/opt/spark/sbin/start-connect-server.sh && tail -f /dev/null"

Connect to Spark

1. Install the Spark ADBC driver:

   dbc install spark --pre

2. Customize the main method in Example.java

   • Change the connection arguments in the params.put() calls
     • Format uri according to the driver documentation, or keep it as is
   • If you changed which database you're connecting to, also change the SQL SELECT statement in stmt.setSqlQuery()

3. Run the Java program:

   mvn compile exec:exec

Clean up

Stop the Docker container running Spark:

docker stop spark-connect