Skip to content

Commit 964f007

Browse files
dlvenablegithub-actions[bot]
dlvenable
authored and
github-actions[bot]
committed
Updated the smoke tests scripts to use the end-to-end tests (#5903)
Updated the smoke tests scripts to use the end-to-end tests to get them running as part of the release again. Support validating tar files by skipping the Docker pull if the image is local. Skip the Docker validations in the run-smoke-tests.sh script altogether to rely on the e2e-tests. Increase validation time to 30 minutes. Signed-off-by: David Venable <dlv@amazon.com> (cherry picked from commit c6f072a)
1 parent 73ff2ba commit 964f007

4 files changed

Lines changed: 16 additions & 182 deletions

File tree

‎.github/workflows/release.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -97,7 +97,7 @@ jobs:
9797
fail-fast: false
9898
runs-on: ubuntu-latest
9999
needs: build
100-
timeout-minutes: 8
100+
timeout-minutes: 30
101101

102102
steps:
103103
- name: Checkout Data Prepper

‎e2e-test/build.gradle‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -85,7 +85,9 @@ subprojects {
8585

8686
tasks.register('dataPrepperDockerImage', DockerProviderTask) {
8787
if(dataPrepperDefinedDockerImage != null) {
88-
dependsOn 'pullDataPrepperDockerImage'
88+
if(dataPrepperDefinedDockerImage.contains('/')) {
89+
dependsOn 'pullDataPrepperDockerImage'
90+
}
8991
imageId = "${dataPrepperDefinedDockerImage}"
9092
}
9193
else if(targetJavaVersion == 'docker') {

‎release/smoke-tests/README.md‎

Lines changed: 5 additions & 71 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,20 @@
11
# Smoke Tests
22

3-
This directory contains smoke tests for Data Prepper. Data Prepper smoke tests perform very basic validation of artifacts to ensure that they were
4-
built correctly. The unit, integration, and end-to-end tests cover for correct functionality.
3+
This directory contains smoke tests for Data Prepper. Data Prepper smoke tests run some of the end-to-end tests.
4+
They are intended to run a small subset of tests on released images to verify they are working.
55

66
## Running smoke tests on a Docker Image
77

88
To run automated smoke test on an image you can use the following command
99

1010
```shell
11-
./release/smoke-tests/run-smoke-tests.sh -v <image_tag> -r <image_repository>
11+
./release/smoke-tests/run-smoke-tests.sh -v <image_tag> -i <image_name>
1212
```
1313

14-
To run smoke tests on the latest published docker image you would run the following command:
14+
To run smoke tests on the latest published Docker image you would run the following command:
1515

1616
```shell
17-
./release/smoke-tests/run-smoke-tests.sh -v latest -r opensearchproject
17+
./release/smoke-tests/run-smoke-tests.sh -v latest -r opensearchproject/data-prepper
1818
```
1919

2020
It is also possible to run smoke tests on a locally built image. Here is an example of targeting a local image `customImageName:myTag`. The image name (-i) is optional, the default value is `opensearch-data-prepper`.
@@ -43,69 +43,3 @@ The values for `-t` are `opensearch-data-prepper` or `opensearch-data-prepper-jd
4343
./release/smoke-tests/run-tarball-files-smoke-tests.sh -i ubuntu:latest -t opensearch-data-prepper-jdk
4444
```
4545

46-
## Troubleshooting smoke tests
47-
48-
If the script is stuck repeating the message "Waiting for Data Prepper to start" try the following steps
49-
50-
### 1. Confirm all containers are running with `docker ps`. Your output should be similar to the following:
51-
```
52-
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
53-
f3d476e2676d smoke-tests_otel-span-exporter "/bin/sh -c 'python3…" 5 minutes ago Restarting (0) 54 seconds ago smoke-tests_otel-span-exporter_1
54-
360db2978df1 otel/opentelemetry-collector:0.40.0 "/otelcol --config=/…" 5 minutes ago Up 5 minutes 0.0.0.0:4317->4317/tcp, :::4317->4317/tcp, 55678-55679/tcp smoke-tests_otel-collector_1
55-
b012352c7593 opensearchproject/data-prepper:latest "/bin/sh -c 'java $J…" 5 minutes ago Up 5 minutes 0.0.0.0:2021->2021/tcp, :::2021->2021/tcp smoke-tests_data-prepper_1
56-
539741e51931 opensearchproject/opensearch:1.0.0 "./opensearch-docker…" 5 minutes ago Up 5 minutes 9300/tcp, 9600/tcp, 0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 9650/tcp node-0.example.com
57-
3b5b1f974174 alpine "/bin/sh -c 'set -x;…" 5 minutes ago Up 5 minutes smoke-tests_http-log-generation_1
58-
```
59-
### 2. Check the container logs, these commands should be executed from ./release/smoke-test.
60-
Tail all container logs:
61-
```
62-
docker-compose logs -f
63-
```
64-
65-
Follow specific container logs:
66-
```
67-
docker-compose logs -f data-prepper
68-
```
69-
70-
The Data Prepper container should show events being processed:
71-
```
72-
...
73-
data-prepper_1 | 2021-12-03T19:57:30,488 [service-map-pipeline-prepper-worker-3-thread-1] INFO org.opensearch.dataprepper.pipeline.ProcessWorker - service-map-pipeline Worker: No records received from buffer
74-
data-prepper_1 | 2021-12-03T19:57:31,925 [grok-pipeline-prepper-worker-7-thread-1] INFO org.opensearch.dataprepper.pipeline.ProcessWorker - grok-pipeline Worker: Processing 8 records from buffer
75-
...
76-
```
77-
### 3. Confirm Open Search is running
78-
```
79-
curl -s -k -u 'admin:<admin password>' 'https://localhost:9200/_cat/indices'
80-
```
81-
If indicies are displayed Open Search is running.
82-
```
83-
yellow open security-auditlog-2021.12.03 8FGpNXmYRamkbuOsdTRORQ 1 1 13 0 75.1kb 75.1kb
84-
yellow open test-grok-index WRuIkpsiQXOcGirpp9hRqw 1 1 77 0 39.1kb 39.1kb
85-
yellow open .opendistro-job-scheduler-lock cyeA9SeST5ibZQPW24CJPA 1 1 1 1 11kb 11kb
86-
yellow open otel-v1-apm-service-map uHdBor9FTSiJDwLAHLroVw 1 1 0 0 208b 208b
87-
green open .opendistro_security IAWYPcinRz66_mamfcLsew 1 0 9 0 57.8kb 57.8kb
88-
yellow open otel-v1-apm-span-000001 3pN9V1YNSjSMY5ggHa2TfA 1 1 12 0 28.3kb 28.3kb
89-
```
90-
**To confirm if Open Search is receiving log data run**
91-
Use the following cURL command to query the index `test-grok-index` for any documents. **Note**: The results may not show all documents received.
92-
```
93-
curl -k -u 'admin:<admin password>' https://localhost:9200/test-grok-index/_search
94-
```
95-
If in your results the JSON path `.hits.total.value` has a value of 0 Open Search is not receiving log data. Confirm with the Data Prepper logs records are being processed from the buffer and no error messages are displayed.
96-
97-
**To confirm if Open Search is receiving trace data run**
98-
Use the following cURL command to query the `otel-v1-apm-span-000001` index for documents containing **PythonService**. These documents will be generated by the **otel-span-exporter**, sent to **otel-collector**, then sent to **data-prepper**, then sent to **opensearch**.
99-
```
100-
curl -k -u 'admin:<admin password>' https://localhost:9200/otel-v1-apm-span-000001/_search?q=PythonService
101-
```
102-
If in your results the JSON path `.hits.total.value` has a value of 0 Open Search is not receiving trace data. Confirm OTel opentelemetry-collector logs are continueally displaying metrics and no errors are printing. Next confirm with the Data Prepper logs records are being processed from the buffer and no error messages are displayed.
103-
104-
### 4. Manually send data to Data Prepper
105-
The following cURL command will send a JSON formatted HTTP log to Data Prepper:
106-
```
107-
curl -k \
108-
-H "Content-Type: application/json" \
109-
-d "[{\"log\": \"smoke test log\"}]" \
110-
"http://localhost:2021/log/ingest"
111-
```

‎release/smoke-tests/run-smoke-tests.sh‎

Lines changed: 7 additions & 109 deletions
Original file line numberDiff line numberDiff line change
@@ -6,23 +6,9 @@
66
set -e
77

88
export IMAGE_NAME="opensearch-data-prepper"
9-
REPO_ROOT=$(git rev-parse --show-toplevel)
10-
export OPENSEARCH_VERSION="1.3.6"
11-
OPENSEARCH_HOST="localhost:9200"
12-
OPENSEARCH_GROK_INDEX="test-grok-index"
13-
OPENSEARCH_OTEL_INDEX="otel-v1-apm-span-000001"
14-
15-
spin[0]="-"
16-
spin[1]="\\"
17-
spin[2]="|"
18-
spin[3]="/"
19-
20-
cd "${REPO_ROOT}/release/smoke-tests"
219

2210
function end_tests () {
2311
local EXIT_CODE=$1
24-
docker-compose down
25-
cd "${REPO_ROOT}"
2612

2713
if [ "${EXIT_CODE}" -ne 0 ]
2814
then
@@ -35,7 +21,7 @@ function end_tests () {
3521

3622
function usage() {
3723
echo ""
38-
echo "This script is used to build the Docker image. It prepares the files required by the Dockerfile in a temporary directory, then builds and tags the Docker image. Script expects to be run from the project root directory."
24+
echo "This script runs specific Data Prepper end-to-end tests to smoke test a release. Script expects to be run from the project root directory."
3925
echo "--------------------------------------------------------------------------"
4026
echo "Usage: $0 [args]"
4127
echo ""
@@ -44,36 +30,10 @@ function usage() {
4430
echo ""
4531
echo "Optional arguments:"
4632
echo -e "-h\t\tPrint this message."
47-
echo -e "-r REPOSITORY\tSpecify the docker repository name (ex: opensearchstaging or opensearchproject). The tag name will be pointed to '-v' value and 'latest'"
4833
echo -e "-i IMAGE_NAME\tOverride the docker image name name (ex: opensearch-data-prepper or data-prepper)."
49-
echo -e "-o OPENSEARCH_VERSION\tOverride the default Open Search version used in smoke tests"
5034
echo "--------------------------------------------------------------------------"
5135
}
5236

53-
function query_hits_gt_zero () {
54-
local URL=$1
55-
local SEARCH_RESPONSE
56-
SEARCH_RESPONSE=$(curl -s -k -u 'admin:myStrongPassword123!' "${URL}")
57-
local LOG_COUNT=0
58-
59-
if command -v jq &> /dev/null
60-
then
61-
LOG_COUNT=$(jq '.hits.total.value' <<< "${SEARCH_RESPONSE}")
62-
else
63-
LOG_COUNT=$(docker run alpine /bin/sh -c "apk -q --no-cache add jq && echo '${SEARCH_RESPONSE}' | jq '.hits.total.value'")
64-
fi
65-
66-
if [ "${LOG_COUNT}" -gt 0 ]
67-
then
68-
echo "Open Search is receiving logs from Data Prepper"
69-
echo "Found at least ${LOG_COUNT} hits"
70-
echo -e "\033[0;32mTest passed\033[0m"
71-
else
72-
echo "No hits found with query url ${URL}"
73-
echo "Smoke test failed"
74-
end_tests 1
75-
fi
76-
}
7737

7838
while getopts "hv:r::o::i::" arg; do
7939
case $arg in
@@ -84,12 +44,6 @@ while getopts "hv:r::o::i::" arg; do
8444
v)
8545
export TAG_NAME=$OPTARG
8646
;;
87-
r)
88-
export REPOSITORY=$OPTARG
89-
;;
90-
o)
91-
export OPENSEARCH_VERSION=$OPTARG
92-
;;
9347
i)
9448
export IMAGE_NAME=$OPTARG
9549
;;
@@ -100,72 +54,16 @@ while getopts "hv:r::o::i::" arg; do
10054
esac
10155
done
10256

103-
if [ -z ${REPOSITORY+x} ]
104-
then
105-
# REPOSITORY not defined
106-
export DOCKER_IMAGE="${IMAGE_NAME}:${TAG_NAME}"
107-
else
108-
# REPOSITORY is defined
109-
export DOCKER_IMAGE="${REPOSITORY}/${IMAGE_NAME}:${TAG_NAME}"
110-
if ! docker pull "${DOCKER_IMAGE}" > /dev/null 2> /dev/null
111-
then
112-
echo "--------------------------------------------------------------------------"
113-
echo "Unable to pull image \"${DOCKER_IMAGE}\" are you sure it exists?"
114-
end_tests 1
115-
fi
116-
fi
117-
118-
if ! docker inspect --type=image "${DOCKER_IMAGE}" > /dev/null
119-
then
120-
echo "--------------------------------------------------------------------------"
121-
echo "Unable to find image \"${DOCKER_IMAGE}\" are you sure it exists?"
122-
end_tests 1
123-
fi
57+
export DOCKER_IMAGE="${IMAGE_NAME}:${TAG_NAME}"
12458

12559
echo "Will smoke test image \"${DOCKER_IMAGE}\""
12660

127-
docker-compose down > /dev/null 2> /dev/null
128-
129-
if ! docker-compose up --detach
130-
then
131-
echo "--------------------------------------------------------------------------"
132-
echo "Failed to start all docker-compose services"
133-
end_tests 1
134-
fi
135-
136-
#Ping Data Prepper until response:
137-
WAITING_FOR_DATAPREPPER=true
138-
echo -n "Waiting for Data Prepper to start "
139-
while ${WAITING_FOR_DATAPREPPER}
140-
do
141-
if curl -s -k -u 'admin:myStrongPassword123!' 'https://localhost:9200/_cat/indices' > /dev/null && curl -s -k -H "Content-Type: application/json" -d '[{"log": "smoke test log "}]' 'http://localhost:2021/log/ingest' > /dev/null
142-
then
143-
WAITING_FOR_DATAPREPPER=false
144-
else
145-
for i in "${spin[@]}"
146-
do
147-
echo -ne "\b$i"
148-
sleep 0.1
149-
done
150-
fi
151-
done
152-
153-
echo -e "\b "
154-
echo "Data Prepper started!"
155-
156-
sleep 30
157-
158-
echo "Ready to begin smoke tests. Running cURL commands."
159-
echo ""
61+
./gradlew -PendToEndDataPrepperImage=${IMAGE_NAME} -PendToEndDataPrepperTag=${TAG_NAME} :e2e-test:log:basicLogEndToEndTest
62+
sleep 1
16063

161-
echo -e "\033[0;33mTest:\033[0m Verify logs received via HTTP were processed"
162-
query_hits_gt_zero "https://${OPENSEARCH_HOST}/${OPENSEARCH_GROK_INDEX}/_search"
163-
echo "Open Search successfully received logs from Data Prepper!"
164-
echo ""
64+
./gradlew -PendToEndDataPrepperImage=${IMAGE_NAME} -PendToEndDataPrepperTag=${TAG_NAME} :e2e-test:trace:rawSpanPeerForwarderEndToEndTest
65+
sleep 1
16566

166-
echo -e "\033[0;33mTest:\033[0m Verify metrics received via grpc were processed"
167-
query_hits_gt_zero "https://${OPENSEARCH_HOST}/${OPENSEARCH_OTEL_INDEX}/_search?q=PythonService"
168-
echo "Open Search successfully received OTel spans from Data Prepper!"
169-
echo ""
67+
./gradlew -PendToEndDataPrepperImage=${IMAGE_NAME} -PendToEndDataPrepperTag=${TAG_NAME} :e2e-test:trace:serviceMapPeerForwarderEndToEndTest
17068

17169
end_tests 0

0 commit comments

Comments
 (0)