Skip to content

[Backport 2.12] Updated the smoke tests scripts to use the end-to-end tests #5913

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dlvenable merged 1 commit into from
Jul 29, 2025
Merged

[Backport 2.12] Updated the smoke tests scripts to use the end-to-end tests #5913

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ jobs:
fail-fast: false
runs-on: ubuntu-latest
needs: build
timeout-minutes: 8
timeout-minutes: 30

steps:
- name: Checkout Data Prepper
Expand Down
4 changes: 3 additions & 1 deletion e2e-test/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,9 @@ subprojects {

tasks.register('dataPrepperDockerImage', DockerProviderTask) {
if(dataPrepperDefinedDockerImage != null) {
dependsOn 'pullDataPrepperDockerImage'
if(dataPrepperDefinedDockerImage.contains('/')) {
dependsOn 'pullDataPrepperDockerImage'
}
imageId = "${dataPrepperDefinedDockerImage}"
}
else if(targetJavaVersion == 'docker') {
Expand Down
76 changes: 5 additions & 71 deletions release/smoke-tests/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
# Smoke Tests

This directory contains smoke tests for Data Prepper. Data Prepper smoke tests perform very basic validation of artifacts to ensure that they were
built correctly. The unit, integration, and end-to-end tests cover for correct functionality.
This directory contains smoke tests for Data Prepper. Data Prepper smoke tests run some of the end-to-end tests.
They are intended to run a small subset of tests on released images to verify they are working.

## Running smoke tests on a Docker Image

To run automated smoke test on an image you can use the following command

```shell
./release/smoke-tests/run-smoke-tests.sh -v <image_tag> -r <image_repository>
./release/smoke-tests/run-smoke-tests.sh -v <image_tag> -i <image_name>
```

To run smoke tests on the latest published docker image you would run the following command:
To run smoke tests on the latest published Docker image you would run the following command:

```shell
./release/smoke-tests/run-smoke-tests.sh -v latest -r opensearchproject
./release/smoke-tests/run-smoke-tests.sh -v latest -r opensearchproject/data-prepper
```

It is also possible to run smoke tests on a locally built image. Here is an example of targeting a local image `customImageName:myTag`. The image name (-i) is optional, the default value is `opensearch-data-prepper`.
Expand Down Expand Up @@ -43,69 +43,3 @@ The values for `-t` are `opensearch-data-prepper` or `opensearch-data-prepper-jd
./release/smoke-tests/run-tarball-files-smoke-tests.sh -i ubuntu:latest -t opensearch-data-prepper-jdk
```

## Troubleshooting smoke tests

If the script is stuck repeating the message "Waiting for Data Prepper to start" try the following steps

### 1. Confirm all containers are running with `docker ps`. Your output should be similar to the following:
```
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f3d476e2676d smoke-tests_otel-span-exporter "/bin/sh -c 'python3…" 5 minutes ago Restarting (0) 54 seconds ago smoke-tests_otel-span-exporter_1
360db2978df1 otel/opentelemetry-collector:0.40.0 "/otelcol --config=/…" 5 minutes ago Up 5 minutes 0.0.0.0:4317->4317/tcp, :::4317->4317/tcp, 55678-55679/tcp smoke-tests_otel-collector_1
b012352c7593 opensearchproject/data-prepper:latest "/bin/sh -c 'java $J…" 5 minutes ago Up 5 minutes 0.0.0.0:2021->2021/tcp, :::2021->2021/tcp smoke-tests_data-prepper_1
539741e51931 opensearchproject/opensearch:1.0.0 "./opensearch-docker…" 5 minutes ago Up 5 minutes 9300/tcp, 9600/tcp, 0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 9650/tcp node-0.example.com
3b5b1f974174 alpine "/bin/sh -c 'set -x;…" 5 minutes ago Up 5 minutes smoke-tests_http-log-generation_1
```
### 2. Check the container logs, these commands should be executed from ./release/smoke-test.
Tail all container logs:
```
docker-compose logs -f
```

Follow specific container logs:
```
docker-compose logs -f data-prepper
```

The Data Prepper container should show events being processed:
```
...
data-prepper_1 | 2021-12-03T19:57:30,488 [service-map-pipeline-prepper-worker-3-thread-1] INFO org.opensearch.dataprepper.pipeline.ProcessWorker - service-map-pipeline Worker: No records received from buffer
data-prepper_1 | 2021-12-03T19:57:31,925 [grok-pipeline-prepper-worker-7-thread-1] INFO org.opensearch.dataprepper.pipeline.ProcessWorker - grok-pipeline Worker: Processing 8 records from buffer
...
```
### 3. Confirm Open Search is running
```
curl -s -k -u 'admin:<admin password>' 'https://localhost:9200/_cat/indices'
```
If indicies are displayed Open Search is running.
```
yellow open security-auditlog-2021.12.03 8FGpNXmYRamkbuOsdTRORQ 1 1 13 0 75.1kb 75.1kb
yellow open test-grok-index WRuIkpsiQXOcGirpp9hRqw 1 1 77 0 39.1kb 39.1kb
yellow open .opendistro-job-scheduler-lock cyeA9SeST5ibZQPW24CJPA 1 1 1 1 11kb 11kb
yellow open otel-v1-apm-service-map uHdBor9FTSiJDwLAHLroVw 1 1 0 0 208b 208b
green open .opendistro_security IAWYPcinRz66_mamfcLsew 1 0 9 0 57.8kb 57.8kb
yellow open otel-v1-apm-span-000001 3pN9V1YNSjSMY5ggHa2TfA 1 1 12 0 28.3kb 28.3kb
```
**To confirm if Open Search is receiving log data run**
Use the following cURL command to query the index `test-grok-index` for any documents. **Note**: The results may not show all documents received.
```
curl -k -u 'admin:<admin password>' https://localhost:9200/test-grok-index/_search
```
If in your results the JSON path `.hits.total.value` has a value of 0 Open Search is not receiving log data. Confirm with the Data Prepper logs records are being processed from the buffer and no error messages are displayed.

**To confirm if Open Search is receiving trace data run**
Use the following cURL command to query the `otel-v1-apm-span-000001` index for documents containing **PythonService**. These documents will be generated by the **otel-span-exporter**, sent to **otel-collector**, then sent to **data-prepper**, then sent to **opensearch**.
```
curl -k -u 'admin:<admin password>' https://localhost:9200/otel-v1-apm-span-000001/_search?q=PythonService
```
If in your results the JSON path `.hits.total.value` has a value of 0 Open Search is not receiving trace data. Confirm OTel opentelemetry-collector logs are continueally displaying metrics and no errors are printing. Next confirm with the Data Prepper logs records are being processed from the buffer and no error messages are displayed.

### 4. Manually send data to Data Prepper
The following cURL command will send a JSON formatted HTTP log to Data Prepper:
```
curl -k \
-H "Content-Type: application/json" \
-d "[{\"log\": \"smoke test log\"}]" \
"http://localhost:2021/log/ingest"
```
116 changes: 7 additions & 109 deletions release/smoke-tests/run-smoke-tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,23 +6,9 @@
set -e

export IMAGE_NAME="opensearch-data-prepper"
REPO_ROOT=$(git rev-parse --show-toplevel)
export OPENSEARCH_VERSION="1.3.6"
OPENSEARCH_HOST="localhost:9200"
OPENSEARCH_GROK_INDEX="test-grok-index"
OPENSEARCH_OTEL_INDEX="otel-v1-apm-span-000001"

spin[0]="-"
spin[1]="\\"
spin[2]="|"
spin[3]="/"

cd "${REPO_ROOT}/release/smoke-tests"

function end_tests () {
local EXIT_CODE=$1
docker-compose down
cd "${REPO_ROOT}"

if [ "${EXIT_CODE}" -ne 0 ]
then
Expand All @@ -35,7 +21,7 @@ function end_tests () {

function usage() {
echo ""
echo "This script is used to build the Docker image. It prepares the files required by the Dockerfile in a temporary directory, then builds and tags the Docker image. Script expects to be run from the project root directory."
echo "This script runs specific Data Prepper end-to-end tests to smoke test a release. Script expects to be run from the project root directory."
echo "--------------------------------------------------------------------------"
echo "Usage: $0 [args]"
echo ""
Expand All @@ -44,36 +30,10 @@ function usage() {
echo ""
echo "Optional arguments:"
echo -e "-h\t\tPrint this message."
echo -e "-r REPOSITORY\tSpecify the docker repository name (ex: opensearchstaging or opensearchproject). The tag name will be pointed to '-v' value and 'latest'"
echo -e "-i IMAGE_NAME\tOverride the docker image name name (ex: opensearch-data-prepper or data-prepper)."
echo -e "-o OPENSEARCH_VERSION\tOverride the default Open Search version used in smoke tests"
echo "--------------------------------------------------------------------------"
}

function query_hits_gt_zero () {
local URL=$1
local SEARCH_RESPONSE
SEARCH_RESPONSE=$(curl -s -k -u 'admin:myStrongPassword123!' "${URL}")
local LOG_COUNT=0

if command -v jq &> /dev/null
then
LOG_COUNT=$(jq '.hits.total.value' <<< "${SEARCH_RESPONSE}")
else
LOG_COUNT=$(docker run alpine /bin/sh -c "apk -q --no-cache add jq && echo '${SEARCH_RESPONSE}' | jq '.hits.total.value'")
fi

if [ "${LOG_COUNT}" -gt 0 ]
then
echo "Open Search is receiving logs from Data Prepper"
echo "Found at least ${LOG_COUNT} hits"
echo -e "\033[0;32mTest passed\033[0m"
else
echo "No hits found with query url ${URL}"
echo "Smoke test failed"
end_tests 1
fi
}

while getopts "hv:r::o::i::" arg; do
case $arg in
Expand All @@ -84,12 +44,6 @@ while getopts "hv:r::o::i::" arg; do
v)
export TAG_NAME=$OPTARG
;;
r)
export REPOSITORY=$OPTARG
;;
o)
export OPENSEARCH_VERSION=$OPTARG
;;
i)
export IMAGE_NAME=$OPTARG
;;
Expand All @@ -100,72 +54,16 @@ while getopts "hv:r::o::i::" arg; do
esac
done

if [ -z ${REPOSITORY+x} ]
then
# REPOSITORY not defined
export DOCKER_IMAGE="${IMAGE_NAME}:${TAG_NAME}"
else
# REPOSITORY is defined
export DOCKER_IMAGE="${REPOSITORY}/${IMAGE_NAME}:${TAG_NAME}"
if ! docker pull "${DOCKER_IMAGE}" > /dev/null 2> /dev/null
then
echo "--------------------------------------------------------------------------"
echo "Unable to pull image \"${DOCKER_IMAGE}\" are you sure it exists?"
end_tests 1
fi
fi

if ! docker inspect --type=image "${DOCKER_IMAGE}" > /dev/null
then
echo "--------------------------------------------------------------------------"
echo "Unable to find image \"${DOCKER_IMAGE}\" are you sure it exists?"
end_tests 1
fi
export DOCKER_IMAGE="${IMAGE_NAME}:${TAG_NAME}"

echo "Will smoke test image \"${DOCKER_IMAGE}\""

docker-compose down > /dev/null 2> /dev/null

if ! docker-compose up --detach
then
echo "--------------------------------------------------------------------------"
echo "Failed to start all docker-compose services"
end_tests 1
fi

#Ping Data Prepper until response:
WAITING_FOR_DATAPREPPER=true
echo -n "Waiting for Data Prepper to start "
while ${WAITING_FOR_DATAPREPPER}
do
if curl -s -k -u 'admin:myStrongPassword123!' 'https://localhost:9200/_cat/indices' > /dev/null && curl -s -k -H "Content-Type: application/json" -d '[{"log": "smoke test log "}]' 'http://localhost:2021/log/ingest' > /dev/null
then
WAITING_FOR_DATAPREPPER=false
else
for i in "${spin[@]}"
do
echo -ne "\b$i"
sleep 0.1
done
fi
done

echo -e "\b "
echo "Data Prepper started!"

sleep 30

echo "Ready to begin smoke tests. Running cURL commands."
echo ""
./gradlew -PendToEndDataPrepperImage=${IMAGE_NAME} -PendToEndDataPrepperTag=${TAG_NAME} :e2e-test:log:basicLogEndToEndTest
sleep 1

echo -e "\033[0;33mTest:\033[0m Verify logs received via HTTP were processed"
query_hits_gt_zero "https://${OPENSEARCH_HOST}/${OPENSEARCH_GROK_INDEX}/_search"
echo "Open Search successfully received logs from Data Prepper!"
echo ""
./gradlew -PendToEndDataPrepperImage=${IMAGE_NAME} -PendToEndDataPrepperTag=${TAG_NAME} :e2e-test:trace:rawSpanPeerForwarderEndToEndTest
sleep 1

echo -e "\033[0;33mTest:\033[0m Verify metrics received via grpc were processed"
query_hits_gt_zero "https://${OPENSEARCH_HOST}/${OPENSEARCH_OTEL_INDEX}/_search?q=PythonService"
echo "Open Search successfully received OTel spans from Data Prepper!"
echo ""
./gradlew -PendToEndDataPrepperImage=${IMAGE_NAME} -PendToEndDataPrepperTag=${TAG_NAME} :e2e-test:trace:serviceMapPeerForwarderEndToEndTest

end_tests 0
Loading