databricks

Connecting Ruby and Databricks with ADBC

Instructions

Prerequisites

1. Install Ruby

2. Install dbc

3. Ensure the native Arrow GLib and ADBC GLib libraries required by red-adbc are installed and discoverable. If bundle install reports missing arrow, arrow-glib, or adbc-glib, use the platform-specific commands below.

   macOS with Homebrew

   brew install apache-arrow-glib apache-arrow-adbc-glib

   Debian/Ubuntu

   sudo apt install libarrow-glib-dev libadbc-glib-dev

   RHEL-compatible distributions

   sudo dnf install arrow-glib-devel adbc-glib-devel

   Windows with RubyInstaller/MSYS2 UCRT64

   pacman -S --needed mingw-w64-ucrt-x86_64-arrow mingw-w64-ucrt-x86_64-arrow-adbc-glib

   If you use a different MSYS2 environment, adjust the package prefix to match it; for example, use mingw-w64-x86_64-* from the MINGW64 shell.

4. Install Ruby dependencies:

   bundle install

   If you have multiple Ruby installations, ensure ruby and bundle resolve to the same installation before running this command.

5. Create a Databricks account or be able to log in to an existing one.

Set up Databricks

1. Log into Databricks and create or locate an existing SQL warehouse.

2. Open the "Connection details" tab and record the server hostname and HTTP path. See the Databricks documentation describing how to get these connection details.

Connect to Databricks

1. Install the Databricks ADBC driver:

   dbc install --level user databricks

2. Customize the Ruby script main.rb:

   • Change the connection arguments in database.set_option():
     • uri is the URI for your Databricks instance. The script includes several authentication options. See the Databricks ADBC driver documentation for details.
   • Change the SQL SELECT statement in connection.query(), or keep it as is.
     • Specify the catalog and schema by fully qualifying the table name as catalog.schema.table.

3. Run the Ruby script:

   bundle exec ruby main.rb