diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index c03dad4e78..b5e394b8c0 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -97,7 +97,7 @@ jobs: fail-fast: false runs-on: ubuntu-latest needs: build - timeout-minutes: 8 + timeout-minutes: 30 steps: - name: Checkout Data Prepper diff --git a/e2e-test/build.gradle b/e2e-test/build.gradle index cf5feed97b..ab1f969605 100644 --- a/e2e-test/build.gradle +++ b/e2e-test/build.gradle @@ -85,7 +85,9 @@ subprojects { tasks.register('dataPrepperDockerImage', DockerProviderTask) { if(dataPrepperDefinedDockerImage != null) { - dependsOn 'pullDataPrepperDockerImage' + if(dataPrepperDefinedDockerImage.contains('/')) { + dependsOn 'pullDataPrepperDockerImage' + } imageId = "${dataPrepperDefinedDockerImage}" } else if(targetJavaVersion == 'docker') { diff --git a/release/smoke-tests/README.md b/release/smoke-tests/README.md index ba175dca79..eea887aca0 100644 --- a/release/smoke-tests/README.md +++ b/release/smoke-tests/README.md @@ -1,20 +1,20 @@ # Smoke Tests -This directory contains smoke tests for Data Prepper. Data Prepper smoke tests perform very basic validation of artifacts to ensure that they were -built correctly. The unit, integration, and end-to-end tests cover for correct functionality. +This directory contains smoke tests for Data Prepper. Data Prepper smoke tests run some of the end-to-end tests. +They are intended to run a small subset of tests on released images to verify they are working. ## Running smoke tests on a Docker Image To run automated smoke test on an image you can use the following command ```shell -./release/smoke-tests/run-smoke-tests.sh -v -r +./release/smoke-tests/run-smoke-tests.sh -v -i ``` -To run smoke tests on the latest published docker image you would run the following command: +To run smoke tests on the latest published Docker image you would run the following command: ```shell -./release/smoke-tests/run-smoke-tests.sh -v latest -r opensearchproject +./release/smoke-tests/run-smoke-tests.sh -v latest -r opensearchproject/data-prepper ``` It is also possible to run smoke tests on a locally built image. Here is an example of targeting a local image `customImageName:myTag`. The image name (-i) is optional, the default value is `opensearch-data-prepper`. @@ -43,69 +43,3 @@ The values for `-t` are `opensearch-data-prepper` or `opensearch-data-prepper-jd ./release/smoke-tests/run-tarball-files-smoke-tests.sh -i ubuntu:latest -t opensearch-data-prepper-jdk ``` -## Troubleshooting smoke tests - -If the script is stuck repeating the message "Waiting for Data Prepper to start" try the following steps - -### 1. Confirm all containers are running with `docker ps`. Your output should be similar to the following: -``` -CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES -f3d476e2676d smoke-tests_otel-span-exporter "/bin/sh -c 'python3…" 5 minutes ago Restarting (0) 54 seconds ago smoke-tests_otel-span-exporter_1 -360db2978df1 otel/opentelemetry-collector:0.40.0 "/otelcol --config=/…" 5 minutes ago Up 5 minutes 0.0.0.0:4317->4317/tcp, :::4317->4317/tcp, 55678-55679/tcp smoke-tests_otel-collector_1 -b012352c7593 opensearchproject/data-prepper:latest "/bin/sh -c 'java $J…" 5 minutes ago Up 5 minutes 0.0.0.0:2021->2021/tcp, :::2021->2021/tcp smoke-tests_data-prepper_1 -539741e51931 opensearchproject/opensearch:1.0.0 "./opensearch-docker…" 5 minutes ago Up 5 minutes 9300/tcp, 9600/tcp, 0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 9650/tcp node-0.example.com -3b5b1f974174 alpine "/bin/sh -c 'set -x;…" 5 minutes ago Up 5 minutes smoke-tests_http-log-generation_1 -``` -### 2. Check the container logs, these commands should be executed from ./release/smoke-test. -Tail all container logs: -``` -docker-compose logs -f -``` - -Follow specific container logs: -``` -docker-compose logs -f data-prepper -``` - -The Data Prepper container should show events being processed: -``` -... -data-prepper_1 | 2021-12-03T19:57:30,488 [service-map-pipeline-prepper-worker-3-thread-1] INFO org.opensearch.dataprepper.pipeline.ProcessWorker - service-map-pipeline Worker: No records received from buffer -data-prepper_1 | 2021-12-03T19:57:31,925 [grok-pipeline-prepper-worker-7-thread-1] INFO org.opensearch.dataprepper.pipeline.ProcessWorker - grok-pipeline Worker: Processing 8 records from buffer -... -``` -### 3. Confirm Open Search is running -``` -curl -s -k -u 'admin:' 'https://localhost:9200/_cat/indices' -``` -If indicies are displayed Open Search is running. -``` -yellow open security-auditlog-2021.12.03 8FGpNXmYRamkbuOsdTRORQ 1 1 13 0 75.1kb 75.1kb -yellow open test-grok-index WRuIkpsiQXOcGirpp9hRqw 1 1 77 0 39.1kb 39.1kb -yellow open .opendistro-job-scheduler-lock cyeA9SeST5ibZQPW24CJPA 1 1 1 1 11kb 11kb -yellow open otel-v1-apm-service-map uHdBor9FTSiJDwLAHLroVw 1 1 0 0 208b 208b -green open .opendistro_security IAWYPcinRz66_mamfcLsew 1 0 9 0 57.8kb 57.8kb -yellow open otel-v1-apm-span-000001 3pN9V1YNSjSMY5ggHa2TfA 1 1 12 0 28.3kb 28.3kb -``` -**To confirm if Open Search is receiving log data run** -Use the following cURL command to query the index `test-grok-index` for any documents. **Note**: The results may not show all documents received. -``` -curl -k -u 'admin:' https://localhost:9200/test-grok-index/_search -``` -If in your results the JSON path `.hits.total.value` has a value of 0 Open Search is not receiving log data. Confirm with the Data Prepper logs records are being processed from the buffer and no error messages are displayed. - -**To confirm if Open Search is receiving trace data run** -Use the following cURL command to query the `otel-v1-apm-span-000001` index for documents containing **PythonService**. These documents will be generated by the **otel-span-exporter**, sent to **otel-collector**, then sent to **data-prepper**, then sent to **opensearch**. -``` -curl -k -u 'admin:' https://localhost:9200/otel-v1-apm-span-000001/_search?q=PythonService -``` -If in your results the JSON path `.hits.total.value` has a value of 0 Open Search is not receiving trace data. Confirm OTel opentelemetry-collector logs are continueally displaying metrics and no errors are printing. Next confirm with the Data Prepper logs records are being processed from the buffer and no error messages are displayed. - -### 4. Manually send data to Data Prepper -The following cURL command will send a JSON formatted HTTP log to Data Prepper: -``` -curl -k \ - -H "Content-Type: application/json" \ - -d "[{\"log\": \"smoke test log\"}]" \ - "http://localhost:2021/log/ingest" -``` diff --git a/release/smoke-tests/run-smoke-tests.sh b/release/smoke-tests/run-smoke-tests.sh index c8d2923f47..05abf194f6 100755 --- a/release/smoke-tests/run-smoke-tests.sh +++ b/release/smoke-tests/run-smoke-tests.sh @@ -6,23 +6,9 @@ set -e export IMAGE_NAME="opensearch-data-prepper" -REPO_ROOT=$(git rev-parse --show-toplevel) -export OPENSEARCH_VERSION="1.3.6" -OPENSEARCH_HOST="localhost:9200" -OPENSEARCH_GROK_INDEX="test-grok-index" -OPENSEARCH_OTEL_INDEX="otel-v1-apm-span-000001" - -spin[0]="-" -spin[1]="\\" -spin[2]="|" -spin[3]="/" - -cd "${REPO_ROOT}/release/smoke-tests" function end_tests () { local EXIT_CODE=$1 - docker-compose down - cd "${REPO_ROOT}" if [ "${EXIT_CODE}" -ne 0 ] then @@ -35,7 +21,7 @@ function end_tests () { function usage() { echo "" - echo "This script is used to build the Docker image. It prepares the files required by the Dockerfile in a temporary directory, then builds and tags the Docker image. Script expects to be run from the project root directory." + echo "This script runs specific Data Prepper end-to-end tests to smoke test a release. Script expects to be run from the project root directory." echo "--------------------------------------------------------------------------" echo "Usage: $0 [args]" echo "" @@ -44,36 +30,10 @@ function usage() { echo "" echo "Optional arguments:" echo -e "-h\t\tPrint this message." - echo -e "-r REPOSITORY\tSpecify the docker repository name (ex: opensearchstaging or opensearchproject). The tag name will be pointed to '-v' value and 'latest'" echo -e "-i IMAGE_NAME\tOverride the docker image name name (ex: opensearch-data-prepper or data-prepper)." - echo -e "-o OPENSEARCH_VERSION\tOverride the default Open Search version used in smoke tests" echo "--------------------------------------------------------------------------" } -function query_hits_gt_zero () { - local URL=$1 - local SEARCH_RESPONSE - SEARCH_RESPONSE=$(curl -s -k -u 'admin:myStrongPassword123!' "${URL}") - local LOG_COUNT=0 - - if command -v jq &> /dev/null - then - LOG_COUNT=$(jq '.hits.total.value' <<< "${SEARCH_RESPONSE}") - else - LOG_COUNT=$(docker run alpine /bin/sh -c "apk -q --no-cache add jq && echo '${SEARCH_RESPONSE}' | jq '.hits.total.value'") - fi - - if [ "${LOG_COUNT}" -gt 0 ] - then - echo "Open Search is receiving logs from Data Prepper" - echo "Found at least ${LOG_COUNT} hits" - echo -e "\033[0;32mTest passed\033[0m" - else - echo "No hits found with query url ${URL}" - echo "Smoke test failed" - end_tests 1 - fi -} while getopts "hv:r::o::i::" arg; do case $arg in @@ -84,12 +44,6 @@ while getopts "hv:r::o::i::" arg; do v) export TAG_NAME=$OPTARG ;; - r) - export REPOSITORY=$OPTARG - ;; - o) - export OPENSEARCH_VERSION=$OPTARG - ;; i) export IMAGE_NAME=$OPTARG ;; @@ -100,72 +54,16 @@ while getopts "hv:r::o::i::" arg; do esac done -if [ -z ${REPOSITORY+x} ] -then - # REPOSITORY not defined - export DOCKER_IMAGE="${IMAGE_NAME}:${TAG_NAME}" -else - # REPOSITORY is defined - export DOCKER_IMAGE="${REPOSITORY}/${IMAGE_NAME}:${TAG_NAME}" - if ! docker pull "${DOCKER_IMAGE}" > /dev/null 2> /dev/null - then - echo "--------------------------------------------------------------------------" - echo "Unable to pull image \"${DOCKER_IMAGE}\" are you sure it exists?" - end_tests 1 - fi -fi - -if ! docker inspect --type=image "${DOCKER_IMAGE}" > /dev/null -then - echo "--------------------------------------------------------------------------" - echo "Unable to find image \"${DOCKER_IMAGE}\" are you sure it exists?" - end_tests 1 -fi +export DOCKER_IMAGE="${IMAGE_NAME}:${TAG_NAME}" echo "Will smoke test image \"${DOCKER_IMAGE}\"" -docker-compose down > /dev/null 2> /dev/null - -if ! docker-compose up --detach -then - echo "--------------------------------------------------------------------------" - echo "Failed to start all docker-compose services" - end_tests 1 -fi - -#Ping Data Prepper until response: -WAITING_FOR_DATAPREPPER=true -echo -n "Waiting for Data Prepper to start " -while ${WAITING_FOR_DATAPREPPER} -do - if curl -s -k -u 'admin:myStrongPassword123!' 'https://localhost:9200/_cat/indices' > /dev/null && curl -s -k -H "Content-Type: application/json" -d '[{"log": "smoke test log "}]' 'http://localhost:2021/log/ingest' > /dev/null - then - WAITING_FOR_DATAPREPPER=false - else - for i in "${spin[@]}" - do - echo -ne "\b$i" - sleep 0.1 - done - fi -done - -echo -e "\b " -echo "Data Prepper started!" - -sleep 30 - -echo "Ready to begin smoke tests. Running cURL commands." -echo "" +./gradlew -PendToEndDataPrepperImage=${IMAGE_NAME} -PendToEndDataPrepperTag=${TAG_NAME} :e2e-test:log:basicLogEndToEndTest +sleep 1 -echo -e "\033[0;33mTest:\033[0m Verify logs received via HTTP were processed" -query_hits_gt_zero "https://${OPENSEARCH_HOST}/${OPENSEARCH_GROK_INDEX}/_search" -echo "Open Search successfully received logs from Data Prepper!" -echo "" +./gradlew -PendToEndDataPrepperImage=${IMAGE_NAME} -PendToEndDataPrepperTag=${TAG_NAME} :e2e-test:trace:rawSpanPeerForwarderEndToEndTest +sleep 1 -echo -e "\033[0;33mTest:\033[0m Verify metrics received via grpc were processed" -query_hits_gt_zero "https://${OPENSEARCH_HOST}/${OPENSEARCH_OTEL_INDEX}/_search?q=PythonService" -echo "Open Search successfully received OTel spans from Data Prepper!" -echo "" +./gradlew -PendToEndDataPrepperImage=${IMAGE_NAME} -PendToEndDataPrepperTag=${TAG_NAME} :e2e-test:trace:serviceMapPeerForwarderEndToEndTest end_tests 0