Skip to content

Commit aab0d13

Browse files
pauloricardomgPaulo Motta
authored andcommitted
CASSSIDECAR-340: Add default process-based lifecycle provider
Implements ProcessLifecycleProvider for managing Cassandra process lifecycle with: - Start and stop operations via sidecar lifecycle APIs - Configurable startup and shutdown commands - Process state tracking and monitoring - Integration tests and example usage documentation Patch by Paulo Motta; reviewed by Arjun Ashok, Francisco Guerrero for CASSSIDECAR-340
1 parent 8ab4167 commit aab0d13

33 files changed

Lines changed: 3375 additions & 27 deletions

File tree

.circleci/config.yml

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,10 @@ jobs:
110110
root: dtest-jars
111111
paths:
112112
- "*.jar"
113+
- persist_to_workspace:
114+
root: cassandra-tarballs
115+
paths:
116+
- "*.tar.gz"
113117

114118
integration_cassandra_40_java11:
115119
docker:
@@ -150,8 +154,10 @@ jobs:
150154
- checkout
151155
- attach_workspace:
152156
at: dtest-jars
157+
- attach_workspace:
158+
at: cassandra-tarballs
153159
- run: ./scripts/install-shaded-dtest-jar-local.sh
154-
- run: ./gradlew --no-daemon -PdtestVersion=4.0.16 -Dcassandra.sidecar.versions_to_test="4.0" integrationTestHeavyWeight --stacktrace
160+
- run: ./gradlew --no-daemon -PdtestVersion=4.0.16 -PtarballVersion=4.0 -Dcassandra.sidecar.versions_to_test="4.0" jar integrationTestHeavyWeight --stacktrace
155161

156162
- store_artifacts:
157163
path: build/reports
@@ -203,8 +209,10 @@ jobs:
203209
- checkout
204210
- attach_workspace:
205211
at: dtest-jars
212+
- attach_workspace:
213+
at: cassandra-tarballs
206214
- run: ./scripts/install-shaded-dtest-jar-local.sh
207-
- run: ./gradlew --no-daemon -PdtestVersion=5.0.3 -Dcassandra.sidecar.versions_to_test="5.0" integrationTestHeavyWeight --stacktrace
215+
- run: ./gradlew --no-daemon -PdtestVersion=5.0.3 -PtarballVersion=5.0 -Dcassandra.sidecar.versions_to_test="5.0" jar integrationTestHeavyWeight --stacktrace
208216

209217
- store_artifacts:
210218
path: build/reports
@@ -256,8 +264,10 @@ jobs:
256264
- checkout
257265
- attach_workspace:
258266
at: dtest-jars
267+
- attach_workspace:
268+
at: cassandra-tarballs
259269
- run: ./scripts/install-shaded-dtest-jar-local.sh
260-
- run: ./gradlew --no-daemon -PdtestVersion=5.1 -Dcassandra.sidecar.versions_to_test="5.1" integrationTestHeavyWeight --stacktrace
270+
- run: ./gradlew --no-daemon -PdtestVersion=5.1 -PtarballVersion=5.1 -Dcassandra.sidecar.versions_to_test="5.1" jar integrationTestHeavyWeight --stacktrace
261271

262272
- store_artifacts:
263273
path: build/reports

.github/workflows/ci.yml

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,9 @@ jobs:
6666
id: cache-dtest-jars
6767
uses: actions/cache@v4
6868
with:
69-
path: dtest-jars/
69+
path: |
70+
dtest-jars/
71+
cassandra-tarballs/
7072
key: dtest-jars-${{ hashFiles('scripts/build-dtest-jars.sh', 'gradle/wrapper/gradle-wrapper.properties') }}
7173

7274
- name: Dtest jars cache status
@@ -89,6 +91,13 @@ jobs:
8991
path: dtest-jars/
9092
retention-days: 90
9193

94+
- name: Upload cassandra tarballs artifact
95+
uses: actions/upload-artifact@v5
96+
with:
97+
name: cassandra-tarballs
98+
path: cassandra-tarballs/
99+
retention-days: 30
100+
92101
# Run unit tests with static analysis on all Java versions
93102
unit-tests:
94103
name: Unit tests (Java ${{ matrix.java }})
@@ -304,6 +313,12 @@ jobs:
304313
name: dtest-jars
305314
path: dtest-jars/
306315

316+
- name: Download cassandra tarballs
317+
uses: actions/download-artifact@v5
318+
with:
319+
name: cassandra-tarballs
320+
path: cassandra-tarballs/
321+
307322
- name: Install dtest jars to local Maven
308323
run: |
309324
./scripts/install-shaded-dtest-jar-local.sh
@@ -314,8 +329,9 @@ jobs:
314329
run: |
315330
./gradlew --no-daemon \
316331
-PdtestVersion=${{ matrix.dtestVersion }} \
332+
-PtarballVersion=${{ matrix.cassandra }} \
317333
-Dcassandra.sidecar.versions_to_test="${{ matrix.cassandra }}" \
318-
integrationTestHeavyWeight \
334+
jar integrationTestHeavyWeight \
319335
--stacktrace
320336
env:
321337
CASSANDRA_DEP_DIR: ${{ github.workspace }}/dtest-jars

.gitignore

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,10 @@ lib
9797
.gradle
9898

9999
dtest-jars
100+
cassandra-tarballs
100101
scripts/dependency-reduced-pom.xml
101102

102103
default-stylesheet.xsl
104+
105+
# Examples
106+
examples/lifecycle/nodes/

CHANGES.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
0.3.0
22
-----
3+
* Add default process-based lifecycle provider (CASSSIDECAR-340)
34
* Sidecar API Endpoint for Nodetool Compaction Stop (CASSSIDECAR-360)
45
* Implementation of CDCPublisher (CASSSIDECAR-243)
56
* Sidecar endpoint for moving a node to a new token (CASSSIDECAR-344)

NEWS.txt

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,12 @@
11
0.3
22
===
33

4+
New features
5+
------------
6+
- ProcessLifecycleProvider for managing Cassandra process lifecycle (start/stop) via sidecar APIs with configurable
7+
startup and shutdown commands, state tracking, and support for custom environment configurations. Usage example
8+
available on examples/lifecycle.
9+
410
0.2
511
===
612

build.gradle

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,7 @@ ext {
9090
}
9191

9292
dependencyLocation = propertyWithDefault("CASSANDRA_DEP_DIR", "${rootDir}/dtest-jars") + "/"
93+
tarballsDir = propertyWithDefault("CASSANDRA_TARBALL_DIR", "${rootDir}/cassandra-tarballs") + "/"
9394

9495
// This allows simplified builds and local maven installs.
9596
forceSigning = propertyExists("forceSigning")
@@ -229,6 +230,7 @@ distributions {
229230
exclude '**.idea/**'
230231
exclude 'bin/**'
231232
exclude 'dtest-jars/**'
233+
exclude 'cassandra-tarballs/**'
232234
exclude '**lib/**'
233235
exclude '**logs/**'
234236
exclude '**/build/**'

client/src/testFixtures/java/org/apache/cassandra/sidecar/client/SidecarClientTest.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,7 @@
9191
import org.apache.cassandra.sidecar.common.response.ListOperationalJobsResponse;
9292
import org.apache.cassandra.sidecar.common.response.ListSnapshotFilesResponse;
9393
import org.apache.cassandra.sidecar.common.response.LiveMigrationStatus;
94+
import org.apache.cassandra.sidecar.common.response.LiveMigrationStatus.MigrationState;
9495
import org.apache.cassandra.sidecar.common.response.NodeSettings;
9596
import org.apache.cassandra.sidecar.common.response.OperationalJobResponse;
9697
import org.apache.cassandra.sidecar.common.response.RingResponse;
@@ -100,7 +101,6 @@
100101
import org.apache.cassandra.sidecar.common.response.TableStatsResponse;
101102
import org.apache.cassandra.sidecar.common.response.TimeSkewResponse;
102103
import org.apache.cassandra.sidecar.common.response.TokenRangeReplicasResponse;
103-
import org.apache.cassandra.sidecar.common.response.LiveMigrationStatus.MigrationState;
104104
import org.apache.cassandra.sidecar.common.response.data.CdcSegmentInfo;
105105
import org.apache.cassandra.sidecar.common.response.data.ClientConnectionEntry;
106106
import org.apache.cassandra.sidecar.common.response.data.CreateRestoreJobResponsePayload;

conf/sidecar.yaml

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,13 @@ cassandra_instances:
6666
jmx_ssl_enabled: false
6767
# jmx_role:
6868
# jmx_role_password:
69+
lifecycle_options:
70+
cassandra_conf_dir: /etc/cassandra
71+
#cassandra_log_dir: /var/log/cassandra
72+
# Optional configuration for the cassandra YAML configuration file location.
73+
# When not specified the configuration file will resolve to
74+
# <lifecycle_options.cassandra_conf_dir>/cassandra.yaml
75+
#cassandra_yaml_path: /etc/cassandra/cassandra.yaml
6976

7077
sidecar:
7178
host: 0.0.0.0
@@ -455,3 +462,16 @@ live_migration:
455462
migration_map: # Map of source and destination Cassandra instances
456463
# localhost1: localhost4 # This entry says that localhost1 will be migrated to localhost4
457464
max_concurrent_downloads: 20 # Maximum number of concurrent downloads allowed
465+
466+
# Configuration to allow sidecar start and stop Cassandra instances via the lifecycle API (disabled by default)
467+
lifecycle:
468+
enabled: false
469+
provider:
470+
class_name: org.apache.cassandra.sidecar.lifecycle.ProcessLifecycleProvider
471+
parameters:
472+
state_dir: /var/lib/cassandra-sidecar/lifecycle # The directory where the process state is stored
473+
cassandra_home: /opt/cassandra # The default Cassandra installation directory
474+
# process_timeout_seconds: 120 # Timeout in seconds for Cassandra process start/stop operations (default: 120)
475+
# sys.cassandra.ring_delay_ms: 30000 # It is possible to set JVM properties via the "sys." prefix
476+
# env.MAX_HEAP_SIZE: 8G # It is possible to set environment variables used to start Cassandra via the "env" prefix
477+
# env.HEAP_NEWSIZE: 8G

examples/lifecycle/README.md

Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
<!--
2+
#
3+
# Licensed to the Apache Software Foundation (ASF) under one
4+
# or more contributor license agreements. See the NOTICE file
5+
# distributed with this work for additional information
6+
# regarding copyright ownership. The ASF licenses this file
7+
# to you under the Apache License, Version 2.0 (the
8+
# "License"); you may not use this file except in compliance
9+
# with the License. You may obtain a copy of the License at
10+
#
11+
# http://www.apache.org/licenses/LICENSE-2.0
12+
#
13+
# Unless required by applicable law or agreed to in writing, software
14+
# distributed under the License is distributed on an "AS IS" BASIS,
15+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
16+
# See the License for the specific language governing permissions and
17+
# limitations under the License.
18+
#
19+
-->
20+
21+
# Starting and stopping Cassandra via lifecycle APIs
22+
23+
In this guide we will show how to start and stop a local Cassandra instance via sidecar lifecycle APIs.
24+
25+
## Pre-requirements
26+
27+
- Configuring Cassandra
28+
- Java 11
29+
30+
## Installing and configuring Cassandra
31+
32+
Use the setup.sh script to install and configure Cassandra and sidecar.
33+
34+
```shell
35+
$ ./setup.sh
36+
```
37+
38+
Once executed, the script should create the following directory structure, simulating a Cassandra host install:
39+
40+
```shell
41+
$ ls -l nodes/localhost/
42+
nodes/localhost/etc:
43+
total 4
44+
drwxr-xr-x 3 paulo paulo 4096 Aug 29 16:58 cassandra
45+
46+
nodes/localhost/opt:
47+
total 4
48+
drwxr-xr-x 8 paulo paulo 4096 Aug 29 16:58 apache-cassandra-4.1.9
49+
50+
nodes/localhost/tmp:
51+
total 50672
52+
-rw-r--r-- 1 paulo paulo 51883388 May 16 08:09 apache-cassandra-4.1.9-bin.tar.gz
53+
54+
nodes/localhost/var:
55+
total 8
56+
drwxr-xr-x 3 paulo paulo 4096 Aug 29 16:58 lib
57+
drwxr-xr-x 3 paulo paulo 4096 Aug 29 16:58 log
58+
```
59+
60+
## Starting sidecar
61+
62+
We can now start our Sidecar instance. The `setup.sh` has already configured `sidecar.yaml` configuration with the correct locations. Start sidecar with:
63+
64+
```shell
65+
./gradlew run -Dsidecar.config=file:///$PWD/examples/lifecycle/conf/sidecar.yaml
66+
```
67+
68+
Since Cassandra is not started yet, you should see the following sidecar logs indicating it's not able to reach Cassandra via JMX:
69+
70+
```
71+
INFO [sidecar-internal-worker-pool-1] 2025-08-29 17:10:33,441 JmxClient.java:197 - Could not connect to JMX on service:jmx:rmi://127.0.0.1:7199/jndi/rmi://127.0.0.1:7199/jmxrmi after 1 attempts. Will retry.
72+
```
73+
74+
## Checking Cassandra lifecycle state via lifecycle API
75+
76+
Use the following command to check that the Cassandra instance is not running and CQL is not up:
77+
78+
```shell
79+
# Check lifecycle state
80+
$ curl localhost:9043/api/v1/cassandra/lifecycle
81+
{"current_state":"STOPPED","desired_state":"UNKNOWN","status":"UNDEFINED","last_update":"No lifecycle task submitted for this instance yet."}
82+
83+
# Check CQL State
84+
$ curl localhost:9043/api/v1/cassandra/native/__health
85+
{"status":"NOT_OK"}
86+
```
87+
88+
## Starting Cassandra via sidecar
89+
90+
Now let's try to start Cassandra:
91+
92+
```shell
93+
$ curl -XPUT http://localhost:9043/api/v1/cassandra/lifecycle -d'{"state": "start"}'
94+
{"current_state":"STOPPED","desired_state":"RUNNING","status":"CONVERGING","last_update":"Submitting start task for instance"}
95+
```
96+
97+
If you see an error during this step, check the logs at `examples/lifecycle/nodes/localhost/var/lib/cassandra-sidecar/lifecycle/start-cassandra-1.out` (and corresponding `start-cassandra-1.err` file).
98+
99+
Query the lifecycle status until the instance is started:
100+
```shell
101+
$ curl localhost:9043/api/v1/cassandra/lifecycle
102+
{"current_state":"RUNNING","desired_state":"RUNNING","status":"CONVERGED","last_update":"Instance has started"}
103+
```
104+
105+
Query the CQL status until it's started. This might take some time since as the Cassandra process initializes.
106+
107+
```shell
108+
$ curl localhost:9043/api/v1/cassandra/native/__health
109+
{"status":"OK"}
110+
```
111+
You should see the following in the sidecar logs, indicating the Cassandra instance is started, and it's able to connect to it via CQL and JMX:
112+
113+
```shell
114+
INFO [sidecar-internal-worker-pool-2] 2025-08-29 17:32:22,504 ProcessLifecycleProvider.java:118 - Starting Cassandra instance localhost with command: [/tmp/examples/lifecycle/nodes/localhost/opt/apache-cassandra-4.1.9/bin/cassandra, -p, /tmp/examples/lifecycle/nodes/localhost/var/lib/cassandra-sidecar/lifecycle/cassandra-localhost.pid, -Dcassandra.ring_delay_ms=5000, -D, cassandra.storagedir=/tmp/examples/lifecycle/nodes/localhost/var/lib/cassandra]
115+
INFO [vert.x-eventloop-thread-2] 2025-08-29 17:32:22,520 ?:? - 0:0:0:0:0:0:0:1 - - [Fri, 29 Aug 2025 21:32:22 GMT] "PUT /api/v1/cassandra/lifecycle HTTP/1.1" 202 126 "-" "curl/8.10.1"
116+
INFO [sidecar-internal-worker-pool-2] 2025-08-29 17:32:25,365 ProcessLifecycleProvider.java:124 - Started Cassandra instance localhost with PID 882
117+
INFO [sidecar-internal-worker-pool-2] 2025-08-29 17:32:48,745 JmxClient.java:215 - Connected to JMX server at service:jmx:rmi://127.0.0.1:7199/jndi/rmi://127.0.0.1:7199/jmxrmi after 1 attempt(s)
118+
INFO [sidecar-internal-worker-pool-2] 2025-08-29 17:32:48,757 CassandraAdapterDelegate.java:225 - Cassandra version change detected (from=null to=4.1.9) for cassandraInstanceId=1. New adapter loaded=CassandraAdapter@694c957d
119+
INFO [sidecar-internal-worker-pool-2] 2025-08-29 17:32:48,758 CassandraAdapterDelegate.java:520 - JMX connected to cassandraInstanceId=1
120+
INFO [sidecar-internal-worker-pool-2] 2025-08-29 17:32:48,758 CQLSessionProviderImpl.java:186 - Connecting to cluster using contact points [/127.0.0.1:9042]
121+
INFO [sidecar-internal-worker-pool-2] 2025-08-29 17:32:48,931 CQLSessionProviderImpl.java:225 - Successfully connected to Cassandra!
122+
INFO [sidecar-internal-worker-pool-2] 2025-08-29 17:32:48,948 CassandraAdapterDelegate.java:529 - CQL connected to cassandraInstanceId=1
123+
INFO [vert.x-eventloop-thread-0] 2025-08-29 17:32:48,951 Server.java:329 - CQL is ready for all Cassandra instances. [1]
124+
```
125+
126+
Check that the Cassandra process ID matches the PID in the lifecycle process ID file:
127+
```shell
128+
$ ps aux | grep CassandraDaemon | grep -v grep | awk '{ print $2 }'
129+
8821
130+
$ cat nodes/localhost/var/lib/cassandra-sidecar/lifecycle/cassandra-1.pid
131+
8821
132+
```
133+
134+
At this stage, you may explore the cassandra logs at `examples/lifecycle/nodes/localhost/var/log/cassandra/system.log` or cassandra startup logs at `examples/lifecycle/nodes/localhost/var/lib/cassandra-sidecar/lifecycle/start-cassandra-1.out`.
135+
136+
## Stopping Cassandra via sidecar
137+
138+
Stop Cassandra via sidecar with the following command:
139+
```shell
140+
$ curl -XPUT http://localhost:9043/api/v1/cassandra/lifecycle -d'{"state": "stop"}'
141+
{"current_state":"RUNNING","desired_state":"STOPPED","status":"CONVERGING","last_update":"Submitting stop task for instance"}
142+
```
143+
144+
If you see an error during this step, check the logs at `examples/lifecycle/nodes/localhost/var/lib/cassandra-sidecar/lifecycle/cassandra-localhost.out` (and corresponding `cassandra-localhost.err` file).
145+
146+
Query the lifecycle status until the process is stopped:
147+
```shell
148+
$ curl http://localhost:9043/api/v1/cassandra/lifecycle
149+
{"current_state":"STOPPED","desired_state":"STOPPED","status":"CONVERGED","last_update":"Instance has stopped"}
150+
```
151+
152+
You should see the following in the sidecar logs, indicating the Cassandra instance is successfully stopped.
153+
154+
```shell
155+
INFO [sidecar-internal-worker-pool-11] 2025-08-29 18:03:05,957 ProcessLifecycleProvider.java:147 - Stopping Cassandra instance localhost with command: [/tmp/examples/lifecycle/nodes/localhost/opt/apache-cassandra-4.1.9/bin/stop-server, -p, /tmp/examples/lifecycle/nodes/localhost/var/lib/cassandra-sidecar/lifecycle/cassandra-localhost.pid]
156+
INFO [vert.x-eventloop-thread-2] 2025-08-29 18:03:05,958 ?:? - 0:0:0:0:0:0:0:1 - - [Fri, 29 Aug 2025 22:03:05 GMT] "PUT /api/v1/cassandra/lifecycle HTTP/1.1" 202 125 "-" "curl/8.10.1"
157+
INFO [sidecar-internal-worker-pool-11] 2025-08-29 18:03:05,960 ProcessLifecycleProvider.java:185 - Waiting for Cassandra instance localhost with PID 15652 to stop...
158+
INFO [cluster11-worker-0] 2025-08-29 18:03:05,969 CassandraAdapterDelegate.java:540 - CQL disconnection from cassandraInstanceId=1
159+
INFO [sidecar-internal-worker-pool-11] 2025-08-29 18:03:10,961 ProcessLifecycleProvider.java:159 - Stopped Cassandra instance localhost with PID 15652.
160+
INFO [sidecar-internal-worker-pool-19] 2025-08-29 18:06:30,985 CassandraAdapterDelegate.java:556 - JMX disconnection from cassandraInstanceId=1
161+
```
162+

0 commit comments

Comments
 (0)