beam-sql.sh, a standalone launcher for Beam SQL Shell by talatuyarer · Pull Request #36305 · apache/beam

talatuyarer · 2025-09-27T07:02:26Z

Currently, using the Beam SQL shell requires cloning the entire Beam repository and running a complex Gradle command with specific project properties just to bundle I/O connectors (e.g., ./gradlew ... -Pbeam.sql.shell.bundled=... installDist). This process is slow on the first run, non-intuitive, and makes it difficult to quickly experiment with different connectors.

This PR introduces beam-sql.sh, a new standalone Bash script that completely automates this process. It provides a simple, powerful, and user-friendly command-line experience for launching the SQL Shell with any required I/O connectors and runners.

Key Features

The ``beam-sql.sh` script is a self-contained utility with the following features:

Dynamic Dependencies: Users can specify the Beam version, I/O connectors (--io), and runners (--runner) directly via command-line flags.
Robust JAR Building: The script dynamically generates a pom.xml and builds a self-contained "uber JAR" using the maven-shade-plugin. It correctly uses the ServicesResourceTransformer to merge META-INF/services files, ensuring all specified connectors are discoverable by the Java ServiceLoader.
Intelligent Caching: Successfully built JARs are cached locally in ~/.beamshell/cache. Subsequent launches with the same configuration are instantaneous, as the script finds and uses the cached JAR, skipping the build step entirely.
No Maven Required: The script makes no assumption about a user's environment. It will use the system-installed mvn if present for speed, but will automatically download and use the Maven Wrapper (mvnw) if mvn is not found, making it highly portable.
Standalone & Easy to Use: It is a single Bash script with minimal, common dependencies (java, curl), making it easy to distribute and use.

How to Use

The script provides a simple and flexible command-line interface.

1. Basic launch with Iceberg support:

./beam-sql.sh --io iceberg

./beamshell --version 2.56.0 --io kafka --io jdbc

gemini-code-assist · 2025-09-27T07:02:41Z

Summary of Changes

Hello @talatuyarer, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces beam-sql.sh, a new standalone Bash script designed to significantly streamline the process of launching the Apache Beam SQL Shell. It eliminates the need for manual repository cloning and complex Gradle commands by automating dependency management, building a self-contained executable JAR with specified I/O connectors and runners, and intelligently caching these builds for rapid subsequent use. The script aims to provide a user-friendly and portable command-line experience for Beam SQL users.

Highlights

Standalone Launcher Script: Introduces beam-sql.sh, a new standalone Bash script that automates the process of launching the Beam SQL Shell, simplifying dependency management and execution.
Dynamic Dependency Management: Users can specify the Beam version, I/O connectors (--io), and runners (--runner) directly via command-line flags, allowing for flexible configuration without manual setup.
Robust JAR Building with Maven Shade Plugin: The script dynamically generates a pom.xml and builds a self-contained 'uber JAR' using the maven-shade-plugin, ensuring proper service loading for all specified connectors.
Intelligent Caching: Successfully built JARs are cached locally in ~/.beamshell/cache. Subsequent launches with the same configuration are instantaneous, as the script reuses the cached JAR.
No Maven Required: The script is highly portable; it will use a system-installed mvn if available, but automatically downloads and uses the Maven Wrapper (mvnw) if mvn is not found.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

github-actions · 2025-09-27T07:34:41Z

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

talatuyarer · 2025-09-27T07:40:33Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces a very useful beam-sql.sh script that significantly simplifies launching the Beam SQL shell. The script is well-designed, with features like dynamic dependency management, caching, and automatic Maven wrapper setup. The accompanying documentation updates are also clear and helpful.

My review focuses on a few areas to improve the long-term maintainability of the script and the documentation. Specifically, I've suggested centralizing hardcoded versions in the script and refactoring the dynamic POM generation for better readability. For the documentation, I've pointed out a few places where hardcoded version numbers could be replaced with more generic text to prevent the docs from becoming stale.

Overall, this is an excellent addition that will greatly improve the user experience for Beam SQL.

gemini-code-assist · 2025-09-27T07:43:10Z

+  # Define URLs for a stable version of the wrapper files
+  local mvnw_script_url="https://raw.githubusercontent.com/apache/maven-wrapper/maven-wrapper-3.2.0/src/main/wrapper/mvnw"
+  local wrapper_jar_url="https://repo.maven.apache.org/maven2/org/apache/maven/wrapper/maven-wrapper/3.2.0/maven-wrapper-3.2.0.jar"
+
+  # We will create the properties file ourselves to specify a modern Maven version
+  echo "distributionUrl=https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.9.6/apache-maven-3.9.6-bin.zip" > "${wrapper_dir}/maven-wrapper.properties"


This block hardcodes URLs and versions for the Maven wrapper and Maven itself. For better maintainability, it's recommended to define these as constants at the top of the script in the Configuration section (lines 24-29). This would also apply to the maven-shade-plugin version on line 206. Centralizing these values makes them easier to find and update.

I agree with gemini :-)

github-actions · 2025-09-27T08:37:11Z

Assigning reviewers:

R: @kennknowles for label website.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

kennknowles

This is great. I added comments for the future. I think removing the "build from source" docs right away is a good idea. We can move them to the README.md in the github repo or something, if they are needed for devs. They were never a good instruction for users.

kennknowles · 2025-09-29T14:28:10Z

+./beam-sql.sh --list-versions
+```
+
+### Option 2: Building from Source


Let's remove this option. Only Beam devs need to know how to build it from souce.

kennknowles · 2025-09-29T14:30:08Z

+DEFAULT_BEAM_VERSION="2.67.0"
+MAIN_CLASS="org.apache.beam.sdk.extensions.sql.jdbc.BeamSqlLine"
+# Directory to store cached executable JAR files
+CACHE_DIR="${HOME}/.beamshell/cache"


Should we just make this .beam so that it can be for anything Beam ends up doing? Or it could be .beam-sql so that it is clearly for this SQL application. I prefer .beam

Done. Make sense

kennknowles · 2025-09-29T14:34:16Z

+  echo ""
+  echo "Options:"
+  echo "  --version   Specify the Apache Beam version (default: ${DEFAULT_BEAM_VERSION})."
+  echo "  --runner    Specify the Beam runner to use (default: direct). Supported: direct, dataflow."


I can already tell that this shell script will eventually graduate to something written in a language that can have a real argument parser :-)

… maintainable

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

chamikaramj

Thanks!

chamikaramj · 2025-09-30T20:58:40Z

+
+```bash
+# Use a specific Beam version
+./beam-sql.sh --version 2.66.0


We should also set this to the latest version during release time. We already do this for documentation.

I dont know how can I do that. Can you help me ? @chamikaramj

From Python you can try this:

beam/sdks/python/apache_beam/yaml/yaml_provider.py

Line 66 in a3fd151

from apache_beam.version import __version__ as beam_version

For documentation we use this:

beam/website/www/site/config.toml

Line 107 in a3fd151

release_latest = "2.68.0"

Both updated during Beam releases.

chamikaramj · 2025-10-02T18:10:29Z

Thanks. LGTM.

ahmedabu98

Looks great overall, just left some comments.

Also, does this support passing in pipeline options (I'm thinking to set up the Dataflow runner)

ahmedabu98 · 2025-10-08T18:55:12Z

+  echo "              Supported runners:"
+  echo "                direct    - DirectRunner (runs locally, good for development)"
+  echo "                dataflow  - DataflowRunner (runs on Google Cloud Dataflow)"
+  echo "  --io        Specify an IO connector to include. Can be used multiple times."


nit: consider making this a comma-separated list for easy use

ahmedabu98 · 2025-10-08T19:00:42Z

+      "prism")
+        echo "  prism            - PrismRunner"
+        echo "                     Local runner for testing portable pipelines."


Are we trying to encourage PrismRunner usage? @damccorm @liferoad

If it's in a good state, maybe it should be listed up higher next to direct runner and dataflow

ahmedabu98 · 2025-10-08T19:05:42Z

+EOL
+# Add IO and Runner dependencies
+  for io in "${IO_CONNECTORS[@]}"; do
+  echo "        <dependency><groupId>org.apache.beam</groupId><artifactId>beam-sdks-java-io-${io}</artifactId><version>\${beam.version}</version></dependency>" >> "${POM_FILE}"


for Iceberg, we'll also need to add beam-sdks-java-extensions-sql-iceberg

These things are resolved by maven I am just created a project which uses necessary dependencies.

talatuyarer · 2025-10-08T19:27:41Z

Looks great overall, just left some comments.

Also, does this support passing in pipeline options (I'm thinking to set up the Dataflow runner)

We dont need to pass pipelineoptions any Pipelineoptions has to be set by SQL SET syntax. Example from documentation:

talatuyarer added 2 commits September 26, 2025 23:43

Initial version of Beam SQL cli

40f1bfe

Added last 10 version list

1b4e44c

Updated Licence Header and add doc for beam-sql.sh

3026a46

github-actions Bot added the website label Sep 27, 2025

gemini-code-assist Bot reviewed Sep 27, 2025

View reviewed changes

github-actions Bot added the Next Action: Reviewers label Sep 27, 2025

kennknowles approved these changes Sep 29, 2025

View reviewed changes

refactoring the Beam SQL shell to make more accessible, reliable, and…

7c500f2

… maintainable

github-actions Bot added build and removed build labels Sep 30, 2025

Apply suggestion from @gemini-code-assist[bot]

dd4242e

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

github-actions Bot added the build label Sep 30, 2025

Apply suggestion from @gemini-code-assist[bot]

6e675a6

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

github-actions Bot added build and removed build labels Sep 30, 2025

liferoad requested a review from ahmedabu98 September 30, 2025 18:30

liferoad assigned chamikaramj Sep 30, 2025

chamikaramj reviewed Sep 30, 2025

View reviewed changes

Addressed Cham comments.

6a91d85

talatuyarer requested a review from chamikaramj October 2, 2025 11:19

ahmedabu98 reviewed Oct 8, 2025

View reviewed changes

chamikaramj merged commit ed39cbb into apache:master Oct 14, 2025
102 of 107 checks passed

Uh oh!

Conversation

talatuyarer commented Sep 27, 2025

Key Features

How to Use

Uh oh!

gemini-code-assist Bot commented Sep 27, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

github-actions Bot commented Sep 27, 2025

Uh oh!

talatuyarer commented Sep 27, 2025

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Sep 27, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Sep 27, 2025

Uh oh!

kennknowles left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chamikaramj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chamikaramj commented Oct 2, 2025

Uh oh!

ahmedabu98 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!