Skip to content

Commit ea291df

Browse files
dlvenabledlvenable
authored
Support building ARM archive files locally using the Gradle build. Resolves #2571. (#6403)
This adds new scripts for running on ARM and includes the ARM architecture in the Gradle release build for linux. Additionally, it updates the smoke tests script to be able to run against different architectures in order to test the changes. Updates the README.md files as well. Also updates the license headers for files in the related projects. Signed-off-by: David Venable <dlv@amazon.com>
1 parent 3bf6c95 commit ea291df

11 files changed

Lines changed: 198 additions & 23 deletions

‎release/archives/README.md‎

Lines changed: 12 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
**Supported Platform**
44

55
* Linux - x64
6+
* Linux - arm64
67

78
### General Instructions
89

@@ -11,21 +12,18 @@
1112
To build all archives, check out the corresponding branch for the version and follow below steps
1213

1314
1. Building an archive depends on the platform on which it will be executed/run
14-
2. For building archives for all supported platforms,
15-
* from root project run `./gradlew :release:archives:buildArchives -Prelease`
16-
* or from current project `gradle buildArchives -Prelease`
15+
2. For building archives for all supported platforms, from root project run `./gradlew :release:archives:buildArchives`
1716
3. Successful build will generate archives in `release/archives/<platform>/build/distributions/` directory
1817
4. Generated archives includes a script file which can be used to execute the data prepper using
1918

2019
```
21-
tar -xzf opendistroforelasticsearch-data-prepper-jdk-v<VERSION>-<PLATFORM>-<ARCHITECTURE>.tar.gz
22-
./opendistroforelasticsearch-data-prepper-jdk-v<VERSION>-<PLATFORM>-<ARCHITECTURE>/data-prepper-tar-install.sh <CONFIG FILE LOCATION>
20+
tar -xzf opensearch-data-prepper-jdk-v<VERSION>-<PLATFORM>-<ARCHITECTURE>.tar.gz
21+
./opensearch-data-prepper-jdk-v<VERSION>-<PLATFORM>-<ARCHITECTURE>/bin/data-prepper <CONFIG FILE LOCATION>
2322
```
2423

2524
#### For platform specific archive
2625

27-
1. From root project, run `./gradlew :release:archives:<platform>:<platform>Tar -Prelease`
28-
* or from relevant platform project `gradle <platform>Tar -Prelease`
26+
1. From root project, run `./gradlew :release:archives:<platform>:<platform>Tar`
2927
2. Successful build generates archives in `release/archives/<platform>/build/distributions/`
3028

3129
##### Example for linux
@@ -35,16 +33,12 @@ For linux, the above steps will be
3533
// from root project
3634
./gradlew :release:archives:linux:linuxTar -Prelease
3735
38-
or
39-
// from linux project
40-
gradle linuxTar -Prelease
41-
4236
cd release/archives/linux/build/distributions/
4337
44-
tar -xzf opendistroforelasticsearch-data-prepper-jdk-<VERSION>-linux-<ARCHITECTURE>.tar.gz
45-
//e.g tar -xzf opendistroforelasticsearch-data-prepper-jdk-0.1.0-linux-x64.tar.gz
38+
tar -xzf opensearch-data-prepper-jdk-<VERSION>-linux-<ARCHITECTURE>.tar.gz
39+
//e.g tar -xzf opensearch-data-prepper-jdk-2.14.0-linux-x64.tar.gz
4640
47-
./opendistroforelasticsearch-data-prepper-jdk-<VERSION>-linux-<ARCHITECTURE>/data-prepper-tar-install.sh <CONFIG FILE LOCATION>
41+
./opensearch-data-prepper-jdk-<VERSION>-linux-<ARCHITECTURE>/bin/data-prepper <CONFIG FILE LOCATION>
4842
4943
```
5044

@@ -57,24 +51,24 @@ Also each platform specific project has a similar mechanism to upload platform s
5751
* From root project, run
5852

5953
```
60-
./gradlew :release:archives:uploadArchives -Prelease -Pregion=us-east-1 -Pbucket=odfe-my-bucket -Pprofile=default
54+
./gradlew :release:archives:uploadArchives -Pregion=us-east-1 -Pbucket=data-prepper-my-bucket -Pprofile=default
6155
6256
or
6357
6458
//below command will use default values for region, bucket and profile
65-
./gradlew :release:archives:uploadArchives -Prelease
59+
./gradlew :release:archives:uploadArchives
6660
```
6761

6862
or
6963

7064
* From current project, run
7165

7266
```
73-
./gradle uploadArchives -Prelease -Pregion=us-east-1 -Pbucket=odfe-my-bucket -Pprofile=default
67+
./gradle uploadArchives -Pregion=us-east-1 -Pbucket=data-prepper-my-bucket -Pprofile=default
7468
7569
or
7670
7771
//below command will use default values for region, bucket and profile
78-
./gradlew uploadArchives -Prelease
72+
./gradlew uploadArchives
7973
8074
```

‎release/archives/build.gradle‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,10 @@
11
/*
22
* Copyright OpenSearch Contributors
33
* SPDX-License-Identifier: Apache-2.0
4+
*
5+
* The OpenSearch Contributors require contributions made to
6+
* this file be licensed under the Apache-2.0 license or a
7+
* compatible open source license.
48
*/
59

610
plugins {

‎release/archives/linux/build.gradle‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,10 @@
11
/*
22
* Copyright OpenSearch Contributors
33
* SPDX-License-Identifier: Apache-2.0
4+
*
5+
* The OpenSearch Contributors require contributions made to
6+
* this file be licensed under the Apache-2.0 license or a
7+
* compatible open source license.
48
*/
59

610
//This file is intentionally left blank to support commands for this project

‎release/archives/linux/data-prepper-arm64.sh‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
#!/bin/bash
2+
3+
#
4+
# Copyright OpenSearch Contributors
5+
# SPDX-License-Identifier: Apache-2.0
6+
#
7+
# The OpenSearch Contributors require contributions made to
8+
# this file be licensed under the Apache-2.0 license or a
9+
# compatible open source license.
10+
#
11+
12+
if [[ $# == 0 ]]; then
13+
echo "Reading pipelines and data-prepper configuration files from Data Prepper home directory."
14+
elif [[ $# == 2 ]]; then
15+
echo
16+
echo "Data Prepper now supports reading pipeline and data-prepper configuration files"
17+
echo "from Data Prepper home directory automatically."
18+
echo "You can continue to specify paths to configuration files as command line arguments,"
19+
echo "but that support will be dropped in a future release."
20+
echo
21+
else
22+
echo
23+
echo "Error: Invalid number of arguments. Expected 0 or 2, received $#."
24+
echo "Put configuration files in Data Prepper home directory and specify no arguments,"
25+
echo "or specify paths to pipeline and data-prepper configuration files, for example:"
26+
echo "bin/data-prepper config/example-pipelines.yaml config/example-data-prepper-config.yaml"
27+
echo
28+
exit 1
29+
fi
30+
31+
MIN_REQ_JAVA_VERSION=11
32+
MIN_REQ_OPENJDK_VERSION=11
33+
DATA_PREPPER_BIN=$(dirname "$(readlink -f "$0")")
34+
DATA_PREPPER_HOME=`readlink -f "$DATA_PREPPER_BIN/.."`
35+
DATA_PREPPER_CLASSPATH="$DATA_PREPPER_HOME/lib/*"
36+
37+
#check if java is installed
38+
if type -p java; then
39+
_java=java
40+
elif [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
41+
_java="$JAVA_HOME/bin/java"
42+
else
43+
echo "java is required for executing data prepper, consider downloading data prepper tar with jdk"
44+
exit 1
45+
fi
46+
47+
if [[ "$_java" ]]
48+
then
49+
java_type=$("$_java" -version 2>&1 | awk -F '"' '/version/ {print $1}')
50+
java_version=$("$_java" -version 2>&1 | awk -F '"' '/version/ {print $2}' | sed 's/\([0-9]*\.[0-9]*\)\..*/\1/g')
51+
echo "Found $java_type of $java_version"
52+
if [[ $java_type == *"openjdk"* ]]
53+
then
54+
if (( $(echo "$java_version < $MIN_REQ_OPENJDK_VERSION" | bc -l) ))
55+
then
56+
echo "Minimum required for $java_type is $MIN_REQ_OPENJDK_VERSION"
57+
exit 1
58+
fi
59+
else
60+
if (( $(echo "$java_version < $MIN_REQ_JAVA_VERSION" | bc -l) ))
61+
then
62+
echo "Minimum required for $java_type is $MIN_REQ_JAVA_VERSION"
63+
exit 1
64+
fi
65+
fi
66+
fi
67+
68+
DATA_PREPPER_HOME_OPTS="-Ddata-prepper.dir=$DATA_PREPPER_HOME"
69+
DATA_PREPPER_JAVA_OPTS="-Dfile.encoding=UTF-8 -Dlog4j.configurationFile=$DATA_PREPPER_HOME/config/log4j2-rolling.properties"
70+
71+
if [[ $# == 0 ]]; then
72+
exec java $DATA_PREPPER_JAVA_OPTS $JAVA_OPTS $DATA_PREPPER_HOME_OPTS -cp "$DATA_PREPPER_CLASSPATH" org.opensearch.dataprepper.DataPrepperExecute
73+
else
74+
PIPELINES_FILE_LOCATION=$1
75+
CONFIG_FILE_LOCATION=$2
76+
exec java $DATA_PREPPER_JAVA_OPTS $JAVA_OPTS $DATA_PREPPER_HOME_OPTS -cp "$DATA_PREPPER_CLASSPATH" org.opensearch.dataprepper.DataPrepperExecute $PIPELINES_FILE_LOCATION $CONFIG_FILE_LOCATION
77+
fi

‎release/archives/linux/data-prepper-jdk-arm64.sh‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
#!/bin/bash
2+
3+
#
4+
# Copyright OpenSearch Contributors
5+
# SPDX-License-Identifier: Apache-2.0
6+
#
7+
# The OpenSearch Contributors require contributions made to
8+
# this file be licensed under the Apache-2.0 license or a
9+
# compatible open source license.
10+
#
11+
12+
if [[ $# == 0 ]]; then
13+
echo "Reading pipelines and data-prepper configuration files from Data Prepper home directory."
14+
elif [[ $# == 2 ]]; then
15+
echo
16+
echo "Data Prepper now supports reading pipeline and data-prepper configuration files"
17+
echo "from Data Prepper home directory automatically."
18+
echo "You can continue to specify paths to configuration files as command line arguments,"
19+
echo "but that support will be dropped in a future release."
20+
echo
21+
else
22+
echo
23+
echo "Error: Invalid number of arguments. Expected 0 or 2, received $#."
24+
echo "Put configuration files in Data Prepper home directory and specify no arguments,"
25+
echo "or specify paths to pipeline and data-prepper configuration files, for example:"
26+
echo "bin/data-prepper config/example-pipelines.yaml config/example-data-prepper-config.yaml"
27+
echo
28+
exit 1
29+
fi
30+
31+
DATA_PREPPER_BIN=$(dirname "$(readlink -f "$0")")
32+
DATA_PREPPER_HOME=`readlink -f "$DATA_PREPPER_BIN/.."`
33+
DATA_PREPPER_CLASSPATH="$DATA_PREPPER_HOME/lib/*"
34+
OPENJDK=$(ls -1 $DATA_PREPPER_HOME/openjdk/ 2>/dev/null)
35+
36+
export JAVA_HOME=$DATA_PREPPER_HOME/openjdk/$OPENJDK
37+
echo "JAVA_HOME is set to $JAVA_HOME"
38+
export PATH=$JAVA_HOME/bin:$PATH
39+
40+
DATA_PREPPER_HOME_OPTS="-Ddata-prepper.dir=$DATA_PREPPER_HOME"
41+
DATA_PREPPER_JAVA_OPTS="-Dfile.encoding=UTF-8 -Dlog4j.configurationFile=$DATA_PREPPER_HOME/config/log4j2-rolling.properties"
42+
43+
if [[ $# == 0 ]]; then
44+
exec java $DATA_PREPPER_JAVA_OPTS $JAVA_OPTS $DATA_PREPPER_HOME_OPTS -cp "$DATA_PREPPER_CLASSPATH" org.opensearch.dataprepper.DataPrepperExecute
45+
else
46+
PIPELINES_FILE_LOCATION=$1
47+
CONFIG_FILE_LOCATION=$2
48+
exec java $DATA_PREPPER_JAVA_OPTS $JAVA_OPTS $DATA_PREPPER_HOME_OPTS -cp "$DATA_PREPPER_CLASSPATH" org.opensearch.dataprepper.DataPrepperExecute $PIPELINES_FILE_LOCATION $CONFIG_FILE_LOCATION
49+
fi

‎release/archives/linux/data-prepper-jdk-x64.sh‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,10 @@
44
# Copyright OpenSearch Contributors
55
# SPDX-License-Identifier: Apache-2.0
66
#
7+
# The OpenSearch Contributors require contributions made to
8+
# this file be licensed under the Apache-2.0 license or a
9+
# compatible open source license.
10+
#
711

812
if [[ $# == 0 ]]; then
913
echo "Reading pipelines and data-prepper configuration files from Data Prepper home directory."

‎release/archives/linux/data-prepper-x64.sh‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,10 @@
44
# Copyright OpenSearch Contributors
55
# SPDX-License-Identifier: Apache-2.0
66
#
7+
# The OpenSearch Contributors require contributions made to
8+
# this file be licensed under the Apache-2.0 license or a
9+
# compatible open source license.
10+
#
711

812
if [[ $# == 0 ]]; then
913
echo "Reading pipelines and data-prepper configuration files from Data Prepper home directory."

‎release/build-resources.gradle‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,20 @@
11
/*
22
* Copyright OpenSearch Contributors
33
* SPDX-License-Identifier: Apache-2.0
4+
*
5+
* The OpenSearch Contributors require contributions made to
6+
* this file be licensed under the Apache-2.0 license or a
7+
* compatible open source license.
48
*/
59

610
//preferably try to main the alphabetical order
711
ext {
812
architectures = [
9-
linux: ['x64']
13+
linux: ['x64', 'arm64']
1014
]
1115
jdkSources = [
1216
linux_x64: 'https://github.com/adoptium/temurin17-binaries/releases/download/jdk-17.0.16%2B8/OpenJDK17U-jdk_x64_linux_hotspot_17.0.16_8.tar.gz',
13-
linux_arm64: 'https://hg.openjdk.java.net/aarch64-port/jdk8/archive/tip.tar.gz'
17+
linux_arm64: 'https://github.com/adoptium/temurin17-binaries/releases/download/jdk-17.0.17%2B10/OpenJDK17U-jdk_aarch64_linux_hotspot_17.0.17_10.tar.gz'
1418
]
1519
awsResources = [
1620
default_region: 'us-east-1',

‎release/smoke-tests/README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,11 +35,13 @@ To run automated smoke test on the default archive file you can use the followin
3535
```
3636

3737
You can also customize what it tests against. The `-i` parameter specifies a base Docker image. The `-t` parameter determines which tar archive file to use.
38-
The values for `-t` are `opensearch-data-prepper` or `opensearch-data-prepper-jdk`.
38+
The values for `-t` are `opensearch-data-prepper` or `opensearch-data-prepper-jdk`. The `-a` parameter specifies the architecture (`x64` or `arm64`, defaults to `x64`).
3939

4040
```shell
4141
./release/smoke-tests/run-tarball-files-smoke-tests.sh -i eclipse-temurin:11 -t opensearch-data-prepper
4242
./release/smoke-tests/run-tarball-files-smoke-tests.sh -i eclipse-temurin:17 -t opensearch-data-prepper
4343
./release/smoke-tests/run-tarball-files-smoke-tests.sh -i ubuntu:latest -t opensearch-data-prepper-jdk
44+
./release/smoke-tests/run-tarball-files-smoke-tests.sh -a arm64 -t opensearch-data-prepper
45+
./release/smoke-tests/run-tarball-files-smoke-tests.sh -a arm64 -t opensearch-data-prepper-jdk -i ubuntu:latest
4446
```
4547

‎release/smoke-tests/data-prepper-tar/build.sh‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,13 @@
11
#!/bin/bash
22

3+
#
34
# Copyright OpenSearch Contributors
45
# SPDX-License-Identifier: Apache-2.0
6+
#
7+
# The OpenSearch Contributors require contributions made to
8+
# this file be licensed under the Apache-2.0 license or a
9+
# compatible open source license.
10+
#
511

612
set -e
713

@@ -31,6 +37,7 @@ if [ -z "${SMOKE_IMAGE_NAME}" ]; then
3137
fi
3238

3339
docker build \
40+
--platform="${PLATFORM:-linux/amd64}" \
3441
--build-arg BUILD_NAME="${BUILD_NAME}" \
3542
--build-arg DATA_PREPPER_VERSION="${DATA_PREPPER_VERSION}" \
3643
--build-arg DOCKER_FILE_DIR="${DOCKER_FILE_DIR}" \

0 commit comments

Comments
 (0)