spark

Connecting C++ and Apache Spark with ADBC

Instructions

Tip

If you already have a Spark instance running, skip the steps to set up and clean up Spark.

Prerequisites

1. Install Pixi

2. Install dbc

Set up Spark

1. Install Docker

2. Start a Spark instance:

   docker run -d --rm --name spark-connect -p 15002:15002 apache/spark:4.1.2 bash -c "/opt/spark/sbin/start-connect-server.sh && tail -f /dev/null"

Connect to Spark

1. Install the Spark ADBC driver:

   dbc install --level user spark --pre

2. Customize the C++ program main.cpp

   • Change the connection arguments in the AdbcDatabaseSetOption() calls
     • Format uri according to the driver documentation, or keep it as is
   • If you changed which database you're connecting to, also change the SQL SELECT statement in AdbcStatementSetSqlQuery()

3. Build and run the C++ program:

   Using Make:

   pixi run make
   ./spark_demo

   Or using CMake:

   pixi run cmake -B build
   pixi run cmake --build build
   ./build/spark_demo

Clean up

1. Stop the Docker container running Spark:

   docker stop spark-connect

2. Clean build artifacts:

   Using Make:

   pixi run make clean

   Using CMake:

   rm -rf build