diff --git a/COMMANDS.md b/COMMANDS.md index 7801ae1d0..db5a18ab9 100644 --- a/COMMANDS.md +++ b/COMMANDS.md @@ -19,6 +19,7 @@ - [Validate Links in Markdown](#validate-links-in-markdown) - [Manual Setup](#manual-setup) - [Setup Neo4j Graph Database](#setup-neo4j-graph-database) + - [Change Neo4j configuration template](#change-neo4j-configuration-template) - [Start Neo4j Graph Database](#start-neo4j-graph-database) - [Setup jQAssistant Java Code Analyzer](#setup-jqassistant-java-code-analyzer) - [Download Maven Artifacts to analyze](#download-maven-artifacts-to-analyze) @@ -51,13 +52,16 @@ ## Start an Analysis -An analysis is started with the script [analyze.sh](./scripts/analysis/analyze.sh). +Before starting an analysis, setup your analysis as described in the [Getting Started](./GETTING_STARTED.md) guide. +An analysis is then started with the script [analyze.sh](./scripts/analysis/analyze.sh). To run all analysis steps simple execute the following command: ```shell ./../../scripts/analysis/analyze.sh ``` +**Hint:** Within the analysis workspace directory you can simply run `analyze.sh` directly without the `../../` prefix since the script is also available in the analysis workspace. + 👉 See [scripts/examples/analyzeAxonFramework.sh](./scripts/examples/analyzeAxonFramework.sh) as an example script that combines all the above steps for a Java Project. 👉 See [scripts/examples/analyzeReactRouter.sh](./scripts/examples/analyzeReactRouter.sh) as an example script that combines all the above steps for a Typescript Project. 👉 See [Code Structure Analysis Pipeline](./.github/workflows/internal-java-code-analysis.yml) on how to do this within a GitHub Actions Workflow. @@ -183,11 +187,23 @@ If any of the script are not allowed to be executed use `chmod +x ./scripts/` fo Use [setupNeo4j.sh](./scripts/setupNeo4j.sh) to download [Neo4j](https://neo4j.com/download-center) and install the plugins [APOC](https://neo4j.com/labs/apoc/4.4) and [Graph Data Science](https://neo4j.com/product/graph-data-science). This script requires the environment variable NEO4J_INITIAL_PASSWORD to be set. It sets the initial password with a temporary `NEO4J_HOME` environment variable to not interfere with a possibly globally installed Neo4j installation. +### Change Neo4j configuration template + +Use [configureNeo4j.sh](./scripts/configureNeo4j.sh) to apply a different Neo4j configuration template from the [scripts/configuration](./scripts/configuration/) directory. This can be useful to optimize Neo4j for different workloads. Example: + +```shell +NEO4J_CONFIG_TEMPLATE=template-neo4j-high-memory.conf ./scripts/configureNeo4j.sh +``` + +**Hint:** In case you want to switch to the high memory profile as in the example, there is a simpler solution. Just run `useNeo4jHighMemoryProfile.sh` from the analysis workspace directory which will set the environment variable `NEO4J_CONFIG_TEMPLATE` and run `configureNeo4j.sh` for you. + ### Start Neo4j Graph Database Use [startNeo4j.sh](./scripts/startNeo4j.sh) to start the locally installed [Neo4j](https://neo4j.com/download-center) Graph database. It runs the script with a temporary `NEO4J_HOME` environment variable to not interfere with a possibly globally installed Neo4j installation. +**Hint:** Within the analysis workspace directory you can simply run `startNeo4j.sh` directly without the `../../` prefix since the script is also available in the analysis workspace. + ### Setup jQAssistant Java Code Analyzer Use [setupJQAssistant.sh](./scripts/setupJQAssistant.sh) to download [jQAssistant](https://jqassistant.github.io/jqassistant/current). @@ -346,6 +362,8 @@ execute_cypher ./cypher/Get_Graph_Data_Science_Library_Version.cypher a=1 Use [stopNeo4j.sh](./scripts/stopNeo4j.sh) to stop the locally running Neo4j Graph Database. It does nothing if the database is already stopped. It runs the script with a temporary `NEO4J_HOME` environment variable to not interfere with a possibly globally installed Neo4j installation. +**Hint:** Within the analysis workspace directory you can run `stopNeo4j.sh` directly without the `../../` prefix since the script is also directly available in the analysis workspace. + ## Jupyter Notebook ### Create a report with executeJupyterNotebookReport.sh diff --git a/README.md b/README.md index a3f694f42..412de9472 100644 --- a/README.md +++ b/README.md @@ -249,7 +249,7 @@ The [Code Structure Analysis Pipeline](./.github/workflows/internal-java-code-an ENABLE_JUPYTER_NOTEBOOK_PDF_GENERATION=true ./../../scripts/analysis/analyze.sh ``` -- How can i disable git log data import? +- How can I disable git log data import? 👉 Set environment variable `IMPORT_GIT_LOG_DATA_IF_SOURCE_IS_PRESENT` to `none`. Example: ```shell @@ -272,34 +272,41 @@ The [Code Structure Analysis Pipeline](./.github/workflows/internal-java-code-an 👉 The custom Jupyter Notebook metadata property `code_graph_analysis_pipeline_data_validation` can be set to choose a query from [cypher/Validation](./cypher/Validation) that will be executed preliminary to the notebook. If the query leads to at least one result, the validation succeeds and the notebook will be run. If the query leads to no result, the notebook will be skipped. For more details see [Data Availability Validation](./COMMANDS.md#data-availability-validation). -- How can i increase the heap memory when scanning large Typescript projects? +- How can I increase the heap memory when scanning large Typescript projects? 👉 Use the environment variable TYPESCRIPT_SCAN_HEAP_MEMORY in megabyte (default = 4096): ```shell TYPESCRIPT_SCAN_HEAP_MEMORY=16384 ./../../scripts/analysis/analyze.sh ``` -- How can i continue on errors when scanning Typescript projects instead of cancelling the whole analysis? +- How can I continue on errors when scanning Typescript projects instead of cancelling the whole analysis? 👉 Use the profile `Neo4j-latest-continue-on-scan-errors` (default = `Neo4j-latest`): ```shell ./../../scripts/analysis/analyze.sh --profile Neo4j-latest-continue-on-scan-errors ``` -- How can i reduce the memory (RAM) consumption? +- How can I reduce the memory (RAM) consumption? 👉 Use the profile `Neo4j-latest-low-memory` (default = `Neo4j-latest`): ```shell ./../../scripts/analysis/analyze.sh --profile Neo4j-latest-low-memory ``` -- How can i increase the memory (RAM) consumption? +- How can I increase the memory (RAM) consumption? 👉 Use the profile `Neo4j-latest-high-memory` (default = `Neo4j-latest`): ```shell ./../../scripts/analysis/analyze.sh --profile Neo4j-latest-high-memory ``` +- How can i increase the memory (RAM) consumption afterwards, when the setup is already done? + 👉 Simply run `useNeo4jHighMemoryProfile.sh` in your analysis working directory, or: + + ```shell + ./../../scripts/useNeo4jHighMemoryProfile.sh + ``` + ## 🕸 Web References - [code-graph-analysis-examples](https://github.com/JohT/code-graph-analysis-examples) diff --git a/init.sh b/init.sh index 7c63ac4df..280e2373a 100755 --- a/init.sh +++ b/init.sh @@ -79,6 +79,7 @@ mkdir -p "./${SOURCE_DIRECTORY}" createForwardingScript "./../../scripts/analysis/analyze.sh" createForwardingScript "./../../scripts/startNeo4j.sh" createForwardingScript "./../../scripts/stopNeo4j.sh" +createForwardingScript "./../../scripts/useNeo4jHighMemoryProfile.sh" source "${SCRIPTS_DIR}/scripts/checkCompatibility.sh" diff --git a/scripts/configuration/template-neo4j-high-memory.conf b/scripts/configuration/template-neo4j-high-memory.conf index 4e204133a..e8df028e7 100644 --- a/scripts/configuration/template-neo4j-high-memory.conf +++ b/scripts/configuration/template-neo4j-high-memory.conf @@ -1,6 +1,3 @@ - -# The following static configuration entries were taken from "template-neo4j.conf". - # Anonymous usage data reporting dbms.usage_report.enabled=false diff --git a/scripts/configuration/template-neo4j-low-memory.conf b/scripts/configuration/template-neo4j-low-memory.conf index 706be2667..2454ab417 100644 --- a/scripts/configuration/template-neo4j-low-memory.conf +++ b/scripts/configuration/template-neo4j-low-memory.conf @@ -1,6 +1,3 @@ - -# The following static configuration entries were taken from "template-neo4j.conf". - # Anonymous usage data reporting dbms.usage_report.enabled=false diff --git a/scripts/configuration/template-neo4j-v4-low-memory.conf b/scripts/configuration/template-neo4j-v4-low-memory.conf index fd5b3b8a1..9a0b1ed0c 100644 --- a/scripts/configuration/template-neo4j-v4-low-memory.conf +++ b/scripts/configuration/template-neo4j-v4-low-memory.conf @@ -1,6 +1,3 @@ - -# The following static configuration entries were taken from "template-neo4j.conf". - # List of procedures and user defined functions that are allowed # full access to the database through unsupported/insecure internal APIs. dbms.security.procedures.unrestricted=apoc.*,gds.* diff --git a/scripts/configuration/template-neo4j-v4.conf b/scripts/configuration/template-neo4j-v4.conf index 8a1200af5..512d3266e 100644 --- a/scripts/configuration/template-neo4j-v4.conf +++ b/scripts/configuration/template-neo4j-v4.conf @@ -1,6 +1,3 @@ - -# The following static configuration entries were taken from "template-neo4j.conf". - # List of procedures and user defined functions that are allowed # full access to the database through unsupported/insecure internal APIs. dbms.security.procedures.unrestricted=apoc.*,gds.* diff --git a/scripts/configuration/template-neo4j.conf b/scripts/configuration/template-neo4j.conf index b4883096b..685faa912 100644 --- a/scripts/configuration/template-neo4j.conf +++ b/scripts/configuration/template-neo4j.conf @@ -1,6 +1,3 @@ - -# The following static configuration entries were taken from "template-neo4j.conf". - # Anonymous usage data reporting dbms.usage_report.enabled=false diff --git a/scripts/configureNeo4j.sh b/scripts/configureNeo4j.sh new file mode 100755 index 000000000..68d7bfdf7 --- /dev/null +++ b/scripts/configureNeo4j.sh @@ -0,0 +1,243 @@ +#!/usr/bin/env bash + +# Configures a (local) Neo4j Community Edition Graph Database (https://neo4j.com/download-center/#community). +# +# Main input is the environment variable NEO4J_CONFIG_TEMPLATE: +# - The name of the template file ("scripts/configuration" folder) for the Neo4j configuration. +# - Defaults to "template-neo4j.conf". +# - The template configuration will be appended to the end of the Neo4j configuration file. +# +# Prerequisites: +# - Intended to be sourced from within another script like the setupNeo4j.sh script +# - Neo4j has been installed +# - The tools directory has been created +# +# Example Usage: +# - NEO4J_CONFIG_TEMPLATE=template-neo4j-high-memory.conf ./../../scripts/configureNeo4j.sh +# +# When run for the first time, the original configuration will be backed up and all configuration entries will be set. +# When run again (re-configuration), the template configuration at the end if the script is executed again. +# +# Requires operatingSystemFunctions.sh + +# Fail on any error ("-e" = exit on first error, "-o pipefail" exist on errors within piped commands) +set -o errexit -o pipefail + +NEO4J_EDITION=${NEO4J_EDITION:-"community"} # Choose "community" or "enterprise" +NEO4J_VERSION=${NEO4J_VERSION:-"2026.01.4"} + +DATA_DIRECTORY=${DATA_DIRECTORY:-"$( pwd -P )/data"} # Path where Neo4j writes its data to (outside tools dir) +RUNTIME_DIRECTORY=${RUNTIME_DIRECTORY:-"$( pwd -P )/runtime"} # Path where Neo4j puts runtime data to (e.g. logs) (outside tools dir) +# Only apply the default tools directory when TOOLS_DIRECTORY is truly unset +# (so that an explicit empty value triggers validation failures in tests). +if [ -z "${TOOLS_DIRECTORY+x}" ]; then + TOOLS_DIRECTORY="tools" +fi +# IMPORT_DIRECTORY default when unset +if [ -z "${IMPORT_DIRECTORY+x}" ]; then + IMPORT_DIRECTORY="$( pwd -P )/import" +fi +# Only apply the default shared downloads directory when truly unset +if [ -z "${SHARED_DOWNLOADS_DIRECTORY+x}" ]; then + SHARED_DOWNLOADS_DIRECTORY="$(dirname "$( pwd )")/downloads" +fi + +NEO4J_HTTP_PORT=${NEO4J_HTTP_PORT:-"7474"} # Neo4j HTTP API port for executing queries +NEO4J_HTTPS_PORT=${NEO4J_HTTPS_PORT:-"7473"} # Neo4j HTTPS port for encrypted querying +NEO4J_BOLT_PORT=${NEO4J_BOLT_PORT:-"7687"} # Neo4j's own "Bolt Protocol" port + +NEO4J_CONFIG_TEMPLATE=${NEO4J_CONFIG_TEMPLATE:-"template-neo4j.conf"} # Name of the template file ("configuration" folder) for the Neo4j configuration. Defaults to "template-neo4j.conf". + +## Get this "scripts" directory if not already set +# Even if $BASH_SOURCE is made for Bourne-like shells it is also supported by others and therefore here the preferred solution. +# CDPATH reduces the scope of the cd command to potentially prevent unintended directory changes. +# This way non-standard tools like readlink aren't needed. +SCRIPTS_DIR=${SCRIPTS_DIR:-$( CDPATH=. cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P )} # Repository directory containing the shell scripts +SCRIPT_NAME="configureNeo4j" +#echo "${SCRIPT_NAME}: SCRIPTS_DIR=${SCRIPTS_DIR}" + +# Internal constants +NEO4J_INSTALLATION_NAME="neo4j-${NEO4J_EDITION}-${NEO4J_VERSION}" +NEO4J_INSTALLATION_DIRECTORY="${TOOLS_DIRECTORY}/${NEO4J_INSTALLATION_NAME}" +NEO4J_CONFIG="${NEO4J_INSTALLATION_DIRECTORY}/conf/neo4j.conf" +NEO4J_MAJOR_VERSION_NUMBER=$(echo "${NEO4J_VERSION}" | cut -d'.' -f1) # First part of the version number (=major version number) +NEO4J_CONFIG_TEMPLATE_PATH="${SCRIPTS_DIR}/configuration/${NEO4J_CONFIG_TEMPLATE}" + +# Include operation system functions like "convertPosixToWindowsPathIfNecessary". +source "${SCRIPTS_DIR}/operatingSystemFunctions.sh" + +fail() { + local ERROR_COLOR='\033[0;31m' # red + local DEFAULT_COLOR='\033[0m' + local errorMessage="${1}" + echo -e "${ERROR_COLOR}${SCRIPT_NAME}: Error: ${errorMessage}${DEFAULT_COLOR}" >&2 + exit 1 +} + +insert_hint_that_the_configuration_is_script_managed() { + local file="$1" + local tempFile + tempFile="$(mktemp)" + + awk ' + BEGIN { + separator_count = 0 + separator_pattern = "^#\\*{11,}$" + } + { + if ($0 ~ separator_pattern) { + separator_count++ + if (separator_count == 2) { + print "#" + print "# This file had been configured with the configureNeo4j.sh script of code-graph-analysis-pipeline." + print "# Manual changes to this file might be overwritten when the script is executed again since the script is designed to update the configuration according to the template configuration." + print "# Please check the documentation of the configureNeo4j.sh script for more details." + } + } + + print + } + ' "${file}" > "${tempFile}" && mv "${tempFile}" "$file" +} + +append_configuration_from_template() { + echo "${SCRIPT_NAME}: Appending configuration template ${NEO4J_CONFIG_TEMPLATE} (memory, procedure permissions, ...)" + local EYE_CATCHER_COMMENT_PREFIX="# The following configuration entries were taken from" + + # Check if a template configuration had already been appended by looking for the eye-catcher comment. + # If it is already there, then delete the old template configuration including the eye-catcher and append the new one again to update it. + # This way this function is idempotent, the configuration is always up to date with the template and there are no duplicate entries. + if grep -q "^${EYE_CATCHER_COMMENT_PREFIX}" "${NEO4J_CONFIG}"; then + echo "${SCRIPT_NAME}: Deleting old configuration that had been appended from the template before since the eye-catcher comment was found. This ensures that there are no duplicate entries and that the configuration is up to date with the template." + sed -i.edit.backup "/^${EYE_CATCHER_COMMENT_PREFIX}/,\$d" "${NEO4J_CONFIG}" + if grep -q '^$' "${NEO4J_CONFIG}"; then + sed -i.edit.backup '$d' "${NEO4J_CONFIG}" # Delete the last line of the config if it is an empty line + fi + rm -f "${NEO4J_CONFIG}.edit.backup" # The backup is created even if not needed to be compatible with both GNU sed and BSD sed (e.g. on macOS) when using -i, but it is removed afterwards to avoid leaving unnecessary files. + fi + + # Append first an eye-catcher comment to the configuration file to indicate that the configuration is script-managed + # and that manual changes might be overwritten. + { + echo "" + echo "${EYE_CATCHER_COMMENT_PREFIX} '${NEO4J_CONFIG_TEMPLATE}' by the script ${SCRIPT_NAME}." + echo "# WARNING: Manual changes below this line might be overwritten when the script is executed again." + echo "" + } >> "${NEO4J_CONFIG}" + + cat "${NEO4J_CONFIG_TEMPLATE_PATH}" >> "${NEO4J_CONFIG}" +} + +NEO4J_INSTALLATION_ADVICE="Please install Neo4j first by running an analysis or by executing the setupNeo4j.sh script directly." + +if [ -z "${TOOLS_DIRECTORY}" ]; then + fail "Requires variable TOOLS_DIRECTORY to be set. If it is the current directory, then use a dot to reflect that." +fi +if [ ! -d "${TOOLS_DIRECTORY}" ]; then + fail "Tools directory <${TOOLS_DIRECTORY}> does not exist. ${NEO4J_INSTALLATION_ADVICE}" +fi + +# Check if SHARED_DOWNLOADS_DIRECTORY variable is set +if [ -z "${SHARED_DOWNLOADS_DIRECTORY}" ]; then + fail "Requires variable SHARED_DOWNLOADS_DIRECTORY to be set. If it is the current directory, then use a dot to reflect that." +fi +if [ ! -d "${SHARED_DOWNLOADS_DIRECTORY}" ]; then + fail "Shared downloads directory <${SHARED_DOWNLOADS_DIRECTORY}> does not exist. ${NEO4J_INSTALLATION_ADVICE}" +fi + +# Fail if Neo4j hadn't been downloaded yet +if [ ! -d "${NEO4J_INSTALLATION_DIRECTORY}" ] ; then + fail "${NEO4J_INSTALLATION_NAME} is not installed into ${TOOLS_DIRECTORY} yet. ${NEO4J_INSTALLATION_ADVICE}" +fi + +if [ ! -f "${NEO4J_CONFIG_TEMPLATE_PATH}" ]; then + fail "Configuration template file ${NEO4J_CONFIG_TEMPLATE} does not exist in the scripts/configuration folder. Please make sure it exists and is correctly named:\n${NEO4J_CONFIG_TEMPLATE_PATH}" +fi + +# Create a backup of the original configuration file before any modification in case there is none yet. +if [ ! -f "${NEO4J_CONFIG}.original.backup" ]; then + cp "${NEO4J_CONFIG}" "${NEO4J_CONFIG}.original.backup" + echo "${SCRIPT_NAME}: First time configuration. Created backup of the original configuration file at ${NEO4J_CONFIG}.original.backup" +else + echo "${SCRIPT_NAME}: Re-Configuration because ${NEO4J_CONFIG}.original.backup already exists." + append_configuration_from_template + echo "${SCRIPT_NAME}: Configuration template replaced successfully." + if [[ "${BASH_SOURCE[0]}" != "$0" ]]; then + return 0 + else + exit 0 + fi +fi + +echo "${SCRIPT_NAME}: Commenting out configuration properties that will later be replaced or are not needed" +if [ "${NEO4J_MAJOR_VERSION_NUMBER}" -ge 5 ]; then + sed -i.edit.backup '/^server\.directories\.import=/ s/^/# defined in the directory section further below #/' "${NEO4J_CONFIG}" + sed -i.edit.backup '/^db\.tx_log\.rotation\.retention_policy=/ s/^/# defined in the transaction section further below #/' "${NEO4J_CONFIG}" +else + sed -i.edit.backup '/^dbms\.directories\.import=/ s/^/# defined in the directory section further below #/' "${NEO4J_CONFIG}" + sed -i.edit.backup '/^dbms\.tx_log\.rotation\.retention_policy=/ s/^/# defined in the transaction section further below #/' "${NEO4J_CONFIG}" +fi +# Remove the edit backup file +rm -f "${NEO4J_CONFIG}.edit.backup" + +# Configure all paths with data that changes (database data, logs, ...) to be in the outside "data" directory +# instead of inside the neo4j directory +echo "${SCRIPT_NAME}: Configuring dynamic settings (data directories, ports, ...)" + +neo4jDataPath=$(convertPosixToWindowsPathIfNecessary "${DATA_DIRECTORY}") +neo4jLogsPath=$(convertPosixToWindowsPathIfNecessary "${RUNTIME_DIRECTORY}/logs") +neo4jDumpsPath=$(convertPosixToWindowsPathIfNecessary "${RUNTIME_DIRECTORY}/dumps") +neo4jRunPath=$(convertPosixToWindowsPathIfNecessary "${RUNTIME_DIRECTORY}/run") +neo4jTransactionsPath=$(convertPosixToWindowsPathIfNecessary "${DATA_DIRECTORY}/transactions") +neo4jImportPath=$(convertPosixToWindowsPathIfNecessary "${IMPORT_DIRECTORY}") + +# Create import directory in case it doesn't exist. +# The import needs to be configured even if its not used since it will be configured below and validated by Neo4j. +mkdir -p "${IMPORT_DIRECTORY}" + +if [ "${NEO4J_MAJOR_VERSION_NUMBER}" -ge 5 ]; then + echo "${SCRIPT_NAME}: Neo4j v5 or higher detected" + { + echo "" + echo "# Paths of data directories in the installation (> v5)" + echo "server.directories.data=${neo4jDataPath}" + echo "server.directories.logs=${neo4jLogsPath}" + echo "server.directories.dumps.root=${neo4jDumpsPath}" + echo "server.directories.run=${neo4jRunPath}" + echo "server.directories.transaction.logs.root=${neo4jTransactionsPath}" + echo "server.directories.import=${neo4jImportPath}" + echo "" + echo "# Ports Configuration (> v5)" + echo "server.bolt.listen_address=:${NEO4J_BOLT_PORT}" + echo "server.bolt.advertised_address=:${NEO4J_BOLT_PORT}" + echo "server.http.listen_address=:${NEO4J_HTTP_PORT}" + echo "server.http.advertised_address=:${NEO4J_HTTP_PORT}" + echo "server.https.listen_address=:${NEO4J_HTTPS_PORT}" + echo "server.https.advertised_address=:${NEO4J_HTTPS_PORT}" + } >> "${NEO4J_CONFIG}" +else + echo "${SCRIPT_NAME}: Neo4j v4 or lower detected" + { + echo "" + echo "# Paths of data directories in the installation" + echo "dbms.directories.data=${neo4jDataPath}" + echo "dbms.directories.logs=${neo4jLogsPath}" + echo "dbms.directories.dumps.root=${neo4jDumpsPath}" + echo "dbms.directories.run=${neo4jRunPath}" +echo "dbms.directories.transaction.logs.root=${neo4jTransactionsPath}" + echo "dbms.directories.import=${neo4jImportPath}" + echo "" + echo "# Ports Configuration" + echo "dbms.connector.bolt.listen_address=:${NEO4J_BOLT_PORT}" + echo "dbms.connector.bolt.advertised_address=:${NEO4J_BOLT_PORT}" + echo "dbms.connector.http.listen_address=:${NEO4J_HTTP_PORT}" + echo "dbms.connector.http.advertised_address=:${NEO4J_HTTP_PORT}" + echo "dbms.connector.https.listen_address=:${NEO4J_HTTPS_PORT}" + echo "dbms.connector.https.advertised_address=:${NEO4J_HTTPS_PORT}" + } >> "${NEO4J_CONFIG}" +fi + +append_configuration_from_template +insert_hint_that_the_configuration_is_script_managed "${NEO4J_CONFIG}" + +echo "${SCRIPT_NAME}: Configuration successfully updated." \ No newline at end of file diff --git a/scripts/markdown/testEmbedMarkdownIncludes.sh b/scripts/markdown/testEmbedMarkdownIncludes.sh index ad4482ea2..a36cce947 100755 --- a/scripts/markdown/testEmbedMarkdownIncludes.sh +++ b/scripts/markdown/testEmbedMarkdownIncludes.sh @@ -26,6 +26,12 @@ successful() { echo -e "testEmbedMarkdownIncludes: ${COLOR_SUCCESSFUL}Tests finished successfully.${COLOR_DEFAULT}" tearDown + # If sourced, return to caller; if executed directly, exit. + if [ "${BASH_SOURCE[0]}" != "$0" ]; then + return 0 + else + exit 0 + fi } fail() { @@ -131,5 +137,4 @@ if [ "${embeddedContent}" != "${expected_test_include_content}" ]; then fi -successful -return 0 \ No newline at end of file +successful \ No newline at end of file diff --git a/scripts/markdown/testFormatQueryResultAsMarkdownTable.sh b/scripts/markdown/testFormatQueryResultAsMarkdownTable.sh index 6a5786573..5e5fca501 100755 --- a/scripts/markdown/testFormatQueryResultAsMarkdownTable.sh +++ b/scripts/markdown/testFormatQueryResultAsMarkdownTable.sh @@ -26,6 +26,12 @@ successful() { echo -e "testFormatQueryResultAsMarkdownTable: ${COLOR_SUCCESSFUL}Tests finished successfully.${COLOR_DEFAULT}" tearDown + # If sourced, return to caller; if executed directly, exit. + if [ "${BASH_SOURCE[0]}" != "$0" ]; then + return 0 + else + exit 0 + fi } fail() { @@ -74,5 +80,4 @@ if [ "${embeddedContent}" != "${expected_result}" ]; then fail "2.) Test failed: Expected Markdown table to be \n${expected_result}, but got:\n${embeddedContent}" fi -successful -return 0 \ No newline at end of file +successful \ No newline at end of file diff --git a/scripts/runTests.sh b/scripts/runTests.sh index e662e6fd1..695d8f64d 100755 --- a/scripts/runTests.sh +++ b/scripts/runTests.sh @@ -17,8 +17,9 @@ LOG_GROUP_END=${LOG_GROUP_END:-"::endgroup::"} # Prefix to end a log group. Defa SCRIPTS_DIR=${SCRIPTS_DIR:-$( CDPATH=. cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P )} # Repository directory containing the shell scripts # echo "runTests: SCRIPTS_DIR=${SCRIPTS_DIR}" >&2 -# Run all report scripts -find "${SCRIPTS_DIR}" -type f -name 'test*.sh' | while read -r test_script_file; do +## Run all test scripts without using a pipe to the while loop so that an +## `exit` inside a sourced test script will terminate this runner (fail-fast). +while IFS= read -r test_script_file; do test_script_filename=$(basename -- "${test_script_file}") test_script_filename="${test_script_filename%.*}" # Remove file extension @@ -29,4 +30,4 @@ find "${SCRIPTS_DIR}" -type f -name 'test*.sh' | while read -r test_script_file; echo "runTests: $(date +'%Y-%m-%dT%H:%M:%S%z') Finished ${test_script_filename}" echo "${LOG_GROUP_END}" -done \ No newline at end of file +done < <(find "${SCRIPTS_DIR}" -type f -name 'test*.sh') \ No newline at end of file diff --git a/scripts/setupNeo4j.sh b/scripts/setupNeo4j.sh index 7e3c2d48f..c5047e706 100755 --- a/scripts/setupNeo4j.sh +++ b/scripts/setupNeo4j.sh @@ -4,7 +4,7 @@ # Note: The environment variable NEO4J_INITIAL_PASSWORD needs to be set. -# Requires download.sh,setupNeo4jInitialPassword.sh,operatingSystemFunctions.sh +# Requires download.sh,configureNeo4j.sh,setupNeo4jInitialPassword.sh,operatingSystemFunctions.sh # Fail on any error ("-e" = exit on first error, "-o pipefail" exist on errors within piped commands) set -o errexit -o pipefail @@ -20,22 +20,12 @@ NEO4J_GDS_PLUGIN_VERSION=${NEO4J_GDS_PLUGIN_VERSION:-"2.26.0"} # Graph Data Scie NEO4J_OPEN_GDS_PLUGIN_VERSION=${NEO4J_OPEN_GDS_PLUGIN_VERSION:-"2.26.0"} # Graph Data Science (GDS) Plugin Version 2.4.x of is compatible with Neo4j 5.x NEO4J_GDS_PLUGIN_EDITION=${NEO4J_GDS_PLUGIN_EDITION:-"open"} # Graph Data Science (GDS) Plugin Edition: "open" for OpenGDS, "full" for the full version with Neo4j license -DATA_DIRECTORY=${DATA_DIRECTORY:-"$( pwd -P )/data"} # Path where Neo4j writes its data to (outside tools dir) -RUNTIME_DIRECTORY=${RUNTIME_DIRECTORY:-"$( pwd -P )/runtime"} # Path where Neo4j puts runtime data to (e.g. logs) (outside tools dir) TOOLS_DIRECTORY=${TOOLS_DIRECTORY:-"tools"} # Get the tools directory (defaults to "tools") -IMPORT_DIRECTORY=${IMPORT_DIRECTORY:-"$( pwd -P )/import"} # The name of the directory that is used to import data (e.g. CSV) files. Defaults to "import". SHARED_DOWNLOADS_DIRECTORY="${SHARED_DOWNLOADS_DIRECTORY:-$(dirname "$( pwd )")/downloads}" -NEO4J_HTTP_PORT=${NEO4J_HTTP_PORT:-"7474"} # Neo4j HTTP API port for executing queries -NEO4J_HTTPS_PORT=${NEO4J_HTTPS_PORT:-"7473"} # Neo4j HTTPS port for encrypted querying -NEO4J_BOLT_PORT=${NEO4J_BOLT_PORT:-"7687"} # Neo4j's own "Bolt Protocol" port - -NEO4J_CONFIG_TEMPLATE=${NEO4J_CONFIG_TEMPLATE:-"template-neo4j.conf"} # Name of the template file ("configuration" folder) for the Neo4j configuration. Defaults to "template-neo4j.conf". - # Internal constants NEO4J_INSTALLATION_NAME="neo4j-${NEO4J_EDITION}-${NEO4J_VERSION}" NEO4J_INSTALLATION_DIRECTORY="${TOOLS_DIRECTORY}/${NEO4J_INSTALLATION_NAME}" -NEO4J_CONFIG="${NEO4J_INSTALLATION_DIRECTORY}/conf/neo4j.conf" NEO4J_PLUGINS="${NEO4J_INSTALLATION_DIRECTORY}/plugins" NEO4J_APOC_CONFIG="${NEO4J_INSTALLATION_DIRECTORY}/conf/apoc.conf" NEO4J_APOC_PLUGIN_ARTIFACT="apoc-${NEO4J_APOC_PLUGIN_VERSION}-${NEO4J_APOC_PLUGIN_EDITION}.jar" @@ -100,76 +90,7 @@ if [ ! -d "${NEO4J_INSTALLATION_DIRECTORY}" ] ; then exit 1 fi - echo "setupNeo4j: Commenting out configuration properties that will later be replaced or are not needed" - if [[ "$NEO4J_MAJOR_VERSION_NUMBER" -ge 5 ]]; then - sed -i.backup '/^server\.directories\.import=/ s/^/# defined in the directory section further below #/' "${NEO4J_CONFIG}" - sed -i.backup '/^db\.tx_log\.rotation\.retention_policy=/ s/^/# defined in the transaction section further below #/' "${NEO4J_CONFIG}" - else - sed -i.backup '/^dbms\.directories\.import=/ s/^/# defined in the directory section further below #/' "${NEO4J_CONFIG}" - sed -i.backup '/^dbms\.tx_log\.rotation\.retention_policy=/ s/^/# defined in the transaction section further below #/' "${NEO4J_CONFIG}" - fi - # Remove the backup file - rm -f "${NEO4J_CONFIG}.backup" - - # Configure all paths with data that changes (database data, logs, ...) to be in the outside "data" directory - # instead of inside the neo4j directory - echo "setupNeo4j: Configuring dynamic settings (data directories, ports, ...)" - - neo4jDataPath=$(convertPosixToWindowsPathIfNecessary "${DATA_DIRECTORY}") - neo4jLogsPath=$(convertPosixToWindowsPathIfNecessary "${RUNTIME_DIRECTORY}/logs") - neo4jDumpsPath=$(convertPosixToWindowsPathIfNecessary "${RUNTIME_DIRECTORY}/dumps") - neo4jRunPath=$(convertPosixToWindowsPathIfNecessary "${RUNTIME_DIRECTORY}/run") - neo4jTransactionsPath=$(convertPosixToWindowsPathIfNecessary "${DATA_DIRECTORY}/transactions") - neo4jImportPath=$(convertPosixToWindowsPathIfNecessary "${IMPORT_DIRECTORY}") - - # Create import directory in case it doesn't exist. - # The import needs to be configured even if its not used since it will be configured below and validated by Neo4j. - mkdir -p "${IMPORT_DIRECTORY}" - - if [[ "$NEO4J_MAJOR_VERSION_NUMBER" -ge 5 ]]; then - echo "setupNeo4j: Neo4j v5 or higher detected" - { - echo "" - echo "# Paths of data directories in the installation (> v5)" - echo "server.directories.data=${neo4jDataPath}" - echo "server.directories.logs=${neo4jLogsPath}" - echo "server.directories.dumps.root=${neo4jDumpsPath}" - echo "server.directories.run=${neo4jRunPath}" - echo "server.directories.transaction.logs.root=${neo4jTransactionsPath}" - echo "server.directories.import=${neo4jImportPath}" - echo "" - echo "# Ports Configuration (> v5)" - echo "server.bolt.listen_address=:${NEO4J_BOLT_PORT}" - echo "server.bolt.advertised_address=:${NEO4J_BOLT_PORT}" - echo "server.http.listen_address=:${NEO4J_HTTP_PORT}" - echo "server.http.advertised_address=:${NEO4J_HTTP_PORT}" - echo "server.https.listen_address=:${NEO4J_HTTPS_PORT}" - echo "server.https.advertised_address=:${NEO4J_HTTPS_PORT}" - } >> "${NEO4J_CONFIG}" - else - echo "setupNeo4j: Neo4j v4 or lower detected" - { - echo "" - echo "# Paths of data directories in the installation" - echo "dbms.directories.data=${neo4jDataPath}" - echo "dbms.directories.logs=${neo4jLogsPath}" - echo "dbms.directories.dumps.root=${neo4jDumpsPath}" - echo "dbms.directories.run=${neo4jRunPath}" - echo "dbms.directories.transaction.logs.root=${neo4jTransactionsPath}" - echo "dbms.directories.import=${neo4jImportPath}" - echo "" - echo "# Ports Configuration" - echo "dbms.connector.bolt.listen_address=:${NEO4J_BOLT_PORT}" - echo "dbms.connector.bolt.advertised_address=:${NEO4J_BOLT_PORT}" - echo "dbms.connector.http.listen_address=:${NEO4J_HTTP_PORT}" - echo "dbms.connector.http.advertised_address=:${NEO4J_HTTP_PORT}" - echo "dbms.connector.https.listen_address=:${NEO4J_HTTPS_PORT}" - echo "dbms.connector.https.advertised_address=:${NEO4J_HTTPS_PORT}" - } >> "${NEO4J_CONFIG}" - fi - - echo "setupNeo4j: Appending configuration template ${NEO4J_CONFIG_TEMPLATE} (memory, procedure permissions, ...)" - cat "${SCRIPTS_DIR}/configuration/${NEO4J_CONFIG_TEMPLATE}" >> "${NEO4J_CONFIG}" + source "${SCRIPTS_DIR}/configureNeo4j.sh" # Set initial password for user "neo4j" otherwise the default password "neo4j" would need to be changed immediately (prompt). # This needs to be done after the configuration changes. @@ -213,7 +134,7 @@ else fi # Download and install the Neo4j plugin "Graph Data Science" (GDS) -if [[ ${NEO4J_GDS_PLUGIN_EDITION} == "open" ]]; then +if [ "${NEO4J_GDS_PLUGIN_EDITION}" == "open" ]; then neo4jGraphDataScienceDownloadUrl="https://github.com/JohT/open-graph-data-science-packaging/releases/download/v${NEO4J_OPEN_GDS_PLUGIN_VERSION}" neo4jGraphDataScienceReleaseArtifact="open-graph-data-science-${NEO4J_OPEN_GDS_PLUGIN_VERSION}-for-neo4j-${NEO4J_MAJOR_VERSION_NUMBER}.jar" else diff --git a/scripts/testCloneGitRepository.sh b/scripts/testCloneGitRepository.sh index 730d0bcb9..55247609a 100755 --- a/scripts/testCloneGitRepository.sh +++ b/scripts/testCloneGitRepository.sh @@ -27,6 +27,13 @@ successful() { echo "" echo -e "${COLOR_DE_EMPHASIZED}${SCRIPT_NAME}:${COLOR_DEFAULT} ${COLOR_SUCCESSFUL}✅ Tests finished successfully.${COLOR_DEFAULT}" tearDown + + # If sourced, return to caller; if executed directly, exit. + if [ "${BASH_SOURCE[0]}" != "$0" ]; then + return 0 + else + exit 0 + fi } info() { @@ -142,5 +149,4 @@ if ! echo "${output}" | grep -q "git clone --bare"; then fail "${test_case_number}.) Test failed: Expected '--bare' option in git clone command." fi -successful -return 0 \ No newline at end of file +successful \ No newline at end of file diff --git a/scripts/testConfigureNeo4j.sh b/scripts/testConfigureNeo4j.sh new file mode 100755 index 000000000..7d0c0c11e --- /dev/null +++ b/scripts/testConfigureNeo4j.sh @@ -0,0 +1,347 @@ +#!/usr/bin/env bash + +# Tests "configureNeo4j.sh". +# +# Usage +# # Run full test suite using a deterministic temporary directory: +# TEST_TMPDIR=./tmp/tmpdir bash scripts/testConfigureNeo4j.sh +# +# # Capture console output to a log file (recommended for CI or debugging): +# TEST_TMPDIR=./tmp/tmpdir bash scripts/testConfigureNeo4j.sh &> test_run.log || true +# +# Troubleshooting +# - To enable a shell trace: run with `bash -x`: +# TEST_TMPDIR=./tmp/tmpdir bash -x scripts/testConfigureNeo4j.sh &> trace.log || true +# - The harness writes per-test logs into the test tempdir as: +# $TEST_TMPDIR/$(basename scripts/testConfigureNeo4j.sh)-.log +# (If you redirect console output to a file, those per-test logs are preserved +# inside the test_run.log capture and can also be inspected directly while the +# test is running.) +# - The test harness will remove the temporary directory on successful completion. +# To preserve artifacts for manual inspection, capture console output to a file +# as shown above (the per-test logs will be visible there), or run the script +# interactively and copy the tempdir path printed in the logs before it is deleted. +# +# Environment / options +# - `TEST_TMPDIR` : location for temporary test workspace (default: system temp) +# - Integration tests run by default; no separate switch required. +# +# Notes +# - The script runs each test in an isolated temp workspace and asserts expected +# success/failure. Use the logs to diagnose failures; grep for "Error:" or +# inspect the per-test log files. + + +# Fail on any error ("-e" = exit on first error, "-o pipefail" exist on errors within piped commands) +set -o errexit -o pipefail + +# Local constants +SCRIPT_NAME=$(basename "${0}") +COLOR_ERROR='\033[0;31m' # red +COLOR_DE_EMPHASIZED='\033[0;90m' # dark gray +COLOR_SUCCESSFUL="\033[0;32m" # green +COLOR_DEFAULT='\033[0m' + +## Get this "scripts" directory if not already set +SCRIPTS_DIR=${SCRIPTS_DIR:-$( CDPATH=. cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P )} + +tearDown() { + rm -rf "${temporaryTestDirectory}" +} + +successful() { + echo "" + echo -e "${COLOR_DE_EMPHASIZED}${SCRIPT_NAME}:${COLOR_DEFAULT} ${COLOR_SUCCESSFUL}✅ Tests finished successfully.${COLOR_DEFAULT}" + tearDown + + # If this script was sourced (by runTests.sh) use `return` so the runner + # continues; if executed directly, use `exit` to terminate the process. + if [ "${BASH_SOURCE[0]}" != "$0" ]; then + return 0 + else + exit 0 + fi +} + +info() { + local infoMessage="${1}" + echo -e "${COLOR_DE_EMPHASIZED}${SCRIPT_NAME}:${COLOR_DEFAULT} ${infoMessage}" +} + +fail() { + local errorMessage="${1}" + echo -e "${COLOR_DE_EMPHASIZED}${SCRIPT_NAME}: ${COLOR_ERROR}${errorMessage}${COLOR_DEFAULT}" + # Keep temporary test directory for inspection and fail fast so CI detects the failure. + if [ -n "${temporaryTestDirectory}" ]; then + echo -e "${COLOR_DE_EMPHASIZED}${SCRIPT_NAME}: Temporary test directory preserved for debugging: ${temporaryTestDirectory}${COLOR_DEFAULT}" + fi + return 1 +} + +printTestLogFileContent() { + local logFile="${temporaryTestDirectory}/${SCRIPT_NAME}-${test_case_number}.log" + if [ ! -f "${logFile}" ]; then + echo -e "${COLOR_DE_EMPHASIZED}(no log produced)${COLOR_DEFAULT}" + return 0 + fi + local logFileContent +logFileContent=$( cat "${logFile}" ) + # Remove color codes from the output for better readability in test logs + logFileContent=$(echo -e "${logFileContent}" | sed -r "s/\x1B\[[0-9;]*[mK]//g") + echo -e "${COLOR_DE_EMPHASIZED}${logFileContent}${COLOR_DEFAULT}" +} + +configureNeo4jExpectingSuccessUnderTest() { + ( + cd "${temporaryTestDirectory}"; + # Export a SCRIPTS_DIR that points to the test's helper stubs so the sourced script will use those + # shellcheck disable=SC2030 + export SCRIPTS_DIR="${temporaryTestDirectory}/scripts" + # Create a small wrapper that exports the test env and sources the configure script. + # Run configureNeo4j.sh in a clean bash via stdin so environment is well-controlled + # Ensure log file exists + logFile="${temporaryTestDirectory}/${SCRIPT_NAME}-${test_case_number}.log" + : >"${logFile}" + # Run configure in a clean bash with controlled env; redirect all output to the log file. + env SCRIPTS_DIR="${temporaryTestDirectory}/scripts" \ + TOOLS_DIRECTORY="${TOOLS_DIRECTORY}" \ + SHARED_DOWNLOADS_DIRECTORY="${SHARED_DOWNLOADS_DIRECTORY}" \ + DATA_DIRECTORY="${DATA_DIRECTORY}" \ + RUNTIME_DIRECTORY="${RUNTIME_DIRECTORY}" \ + IMPORT_DIRECTORY="${IMPORT_DIRECTORY}" \ + bash -c "exec >'${logFile}' 2>&1; source '${REPO_CONFIGURE_SCRIPT}'" + ) + exitCode=$? + if [ ${exitCode} -ne 0 ]; then + fail "❌ Test failed: Script exited with non-zero exit code ${exitCode}." + fi + printTestLogFileContent +} + +configureNeo4jExpectingFailureUnderTest() { + set +o errexit + ( + cd "${temporaryTestDirectory}"; + # shellcheck disable=SC2031 + export SCRIPTS_DIR="${temporaryTestDirectory}/scripts" + # Ensure log file exists + logFile="${temporaryTestDirectory}/${SCRIPT_NAME}-${test_case_number}.log" + : >"${logFile}" + env SCRIPTS_DIR="${temporaryTestDirectory}/scripts" \ + TOOLS_DIRECTORY="${TOOLS_DIRECTORY}" \ + SHARED_DOWNLOADS_DIRECTORY="${SHARED_DOWNLOADS_DIRECTORY}" \ + DATA_DIRECTORY="${DATA_DIRECTORY}" \ + RUNTIME_DIRECTORY="${RUNTIME_DIRECTORY}" \ + IMPORT_DIRECTORY="${IMPORT_DIRECTORY}" \ + bash -c "exec >'${logFile}' 2>&1; source '${REPO_CONFIGURE_SCRIPT}'" + ) + exitCode=$? + set -o errexit + if [ ${exitCode} -eq 0 ]; then + fail "❌ Test failed: Script exited with zero exit code but was expected to fail." + fi + printTestLogFileContent +} + +info "Starting tests...." + +# Create testing resources +temporaryTestDirectory=${TEST_TMPDIR:-$(mktemp -d 2>/dev/null || mktemp -d -t "temporaryTestDirectory_${SCRIPT_NAME}")} +if [ -n "${TEST_TMPDIR}" ]; then + mkdir -p "${temporaryTestDirectory}" +fi +# Normalize to absolute path to avoid relative-path duplication when cd'ing into it +temporaryTestDirectory=$(cd "${temporaryTestDirectory}" && pwd -P) + +# The test will source the repository's configureNeo4j.sh but override SCRIPTS_DIR so that the script +# picks up test-provided helper files (operatingSystemFunctions.sh) and template files from the temp dir. +# Compute repository scripts dir explicitly and path to the configure script. +REPO_SCRIPTS_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P) +REPO_CONFIGURE_SCRIPT="${REPO_SCRIPTS_DIR}/configureNeo4j.sh" + +mkdir -p "${temporaryTestDirectory}/scripts/configuration" + +# Use the real operatingSystemFunctions.sh from the repo but disable its +# interactive/stderr print (printWindows) to avoid noisy output during tests. +if [ -r "${REPO_SCRIPTS_DIR}/operatingSystemFunctions.sh" ]; then + mkdir -p "${temporaryTestDirectory}/scripts" + if sed -e 's/^printWindows\s*$/#printWindows/' "${REPO_SCRIPTS_DIR}/operatingSystemFunctions.sh" > "${temporaryTestDirectory}/scripts/operatingSystemFunctions.sh" 2>/dev/null; then + : + else + # fallback to a minimal copy if sed failed for some reason + cp "${REPO_SCRIPTS_DIR}/operatingSystemFunctions.sh" "${temporaryTestDirectory}/scripts/operatingSystemFunctions.sh" 2>/dev/null || true + fi +else + # Fallback: minimal implementation if the repo file is missing + cat > "${temporaryTestDirectory}/scripts/operatingSystemFunctions.sh" <<'EOF' +#!/usr/bin/env bash +convertPosixToWindowsPathIfNecessary() { echo "$1"; } +EOF +fi + +# Helper to create a minimal neo4j.conf for tests with two separator lines so insert_hint is exercised +create_base_neo4j_configuration() { + local configurationPath="$1" + mkdir -p "$(dirname "${configurationPath}")" + cat > "${configurationPath}" <<'EOF' +#************* +# Base configuration for tests +#************* +# Some existing entry +dbms.memory.pagecache.size=512M +EOF +} + +## Run integration + validation tests by default. + +# ------- Integration Test Case (v5) +test_case_number=1 +echo "" +info "${test_case_number}.) it should append template and update neo4j config for Neo4j v5." + +mkdir -p "${temporaryTestDirectory}/tools" +mkdir -p "${temporaryTestDirectory}/downloads" + +# Provide minimal neo4j installation dir expected by the script +NEO4J_INSTALLATION_NAME="neo4j-community-2026.01.4" + mkdir -p "${temporaryTestDirectory}/tools/${NEO4J_INSTALLATION_NAME}/conf" + create_base_neo4j_configuration "${temporaryTestDirectory}/tools/${NEO4J_INSTALLATION_NAME}/conf/neo4j.conf" + +# Create template with minimal entries expected by tests (v5 entries) +cat > "${temporaryTestDirectory}/scripts/configuration/template-neo4j.conf" <<'EOF' +# Template for tests +server.directories.import=/some/import/path +db.tx_log.rotation.retention_policy=7 days +# End of configuration from template-neo4j.conf +EOF + +# Export env so that the sourced script uses the test directories +export TOOLS_DIRECTORY="${temporaryTestDirectory}/tools" +export SHARED_DOWNLOADS_DIRECTORY="${temporaryTestDirectory}/downloads" +export DATA_DIRECTORY="${temporaryTestDirectory}/data" +export RUNTIME_DIRECTORY="${temporaryTestDirectory}/runtime" +export IMPORT_DIRECTORY="${temporaryTestDirectory}/import" + +configureNeo4jExpectingSuccessUnderTest + +# Validate that backup had been created and template entries appended +if [ ! -f "${temporaryTestDirectory}/tools/${NEO4J_INSTALLATION_NAME}/conf/neo4j.conf.original.backup" ]; then + fail "${test_case_number}.) Test failed: Expected original backup file not found." +fi +if ! grep -q "server.directories.import=" "${temporaryTestDirectory}/tools/${NEO4J_INSTALLATION_NAME}/conf/neo4j.conf"; then + fail "${test_case_number}.) Test failed: Expected 'server.directories.import' entry not found in neo4j.conf." +fi +if ! grep -q "server.bolt.listen_address" "${temporaryTestDirectory}/tools/${NEO4J_INSTALLATION_NAME}/conf/neo4j.conf"; then + fail "${test_case_number}.) Test failed: Expected 'server.bolt.listen_address' entry not found in neo4j.conf (v5 entries)." +fi + +# ------- Failure Cases: missing or empty TOOLS_DIRECTORY +test_case_number=2 +echo "" +info "${test_case_number}.) it should fail when TOOLS_DIRECTORY is unset or empty." + +export TOOLS_DIRECTORY="" +configureNeo4jExpectingFailureUnderTest + +# ------- Failure Cases: TOOLS_DIRECTORY does not exist +test_case_number=3 +echo "" +info "${test_case_number}.) it should fail when TOOLS_DIRECTORY does not exist." + +export TOOLS_DIRECTORY="${temporaryTestDirectory}/tools_missing" +configureNeo4jExpectingFailureUnderTest + +# ------- Failure Cases: missing or empty SHARED_DOWNLOADS_DIRECTORY +test_case_number=4 +echo "" +info "${test_case_number}.) it should fail when SHARED_DOWNLOADS_DIRECTORY is unset or empty." + +export TOOLS_DIRECTORY="${temporaryTestDirectory}/tools" +export SHARED_DOWNLOADS_DIRECTORY="" +configureNeo4jExpectingFailureUnderTest + +# ------- Failure Cases: SHARED_DOWNLOADS_DIRECTORY does not exist +test_case_number=5 +echo "" +info "${test_case_number}.) it should fail when SHARED_DOWNLOADS_DIRECTORY does not exist." + +export SHARED_DOWNLOADS_DIRECTORY="${temporaryTestDirectory}/missing_downloads_dir" +configureNeo4jExpectingFailureUnderTest + +# ------- Failure Case: Neo4j installation dir missing +test_case_number=6 +echo "" +info "${test_case_number}.) it should fail when neo4j installation directory is missing." + +# create tools dir but remove installation dir +mkdir -p "${temporaryTestDirectory}/tools" +rm -rf "${temporaryTestDirectory}/tools/${NEO4J_INSTALLATION_NAME}" +export SHARED_DOWNLOADS_DIRECTORY="${temporaryTestDirectory}/downloads" +configureNeo4jExpectingFailureUnderTest + +# ------- Failure Case: configuration template missing +test_case_number=7 +echo "" +info "${test_case_number}.) it should fail when the configuration template file is missing." + +# Recreate the installation and conf + mkdir -p "${temporaryTestDirectory}/tools/${NEO4J_INSTALLATION_NAME}/conf" + create_base_neo4j_configuration "${temporaryTestDirectory}/tools/${NEO4J_INSTALLATION_NAME}/conf/neo4j.conf" +rm -f "${temporaryTestDirectory}/scripts/configuration/${NEO4J_CONFIG_TEMPLATE:-template-neo4j.conf}" +configureNeo4jExpectingFailureUnderTest + +# ------- Integration Test Case: re-configuration path (backup exists) +test_case_number=8 +echo "" +info "${test_case_number}.) it should re-configure (append/replace template) when original backup exists." + +# Create template again +cat > "${temporaryTestDirectory}/scripts/configuration/template-neo4j.conf" <<'EOF' +# Template for re-configuration test +server.directories.import=/another/import +db.tx_log.rotation.retention_policy=14 days +# End of configuration from template-neo4j.conf +EOF + +# Ensure original backup file exists to trigger reconfiguration branch +touch "${temporaryTestDirectory}/tools/${NEO4J_INSTALLATION_NAME}/conf/neo4j.conf.original.backup" + +export TOOLS_DIRECTORY="${temporaryTestDirectory}/tools" +export SHARED_DOWNLOADS_DIRECTORY="${temporaryTestDirectory}/downloads" +export DATA_DIRECTORY="${temporaryTestDirectory}/data" +export RUNTIME_DIRECTORY="${temporaryTestDirectory}/runtime" +export IMPORT_DIRECTORY="${temporaryTestDirectory}/import" + +configureNeo4jExpectingSuccessUnderTest + +if ! grep -q "The following configuration entries were taken from" "${temporaryTestDirectory}/tools/${NEO4J_INSTALLATION_NAME}/conf/neo4j.conf"; then + fail "${test_case_number}.) Test failed: Eye-catcher comment from template append not found." +fi + +# ------- Small check for Neo4j v4 related formatting (unit) +test_case_number=9 +echo "" +info "${test_case_number}.) it should write v4 style properties when NEO4J_VERSION is v4 (minimal check)." + +# Prepare environment for v4 run +export NEO4J_VERSION="4.4.0" +export TOOLS_DIRECTORY="${temporaryTestDirectory}/tools" +mkdir -p "${temporaryTestDirectory}/tools/neo4j-community-4.4.0/conf" + create_base_neo4j_configuration "${temporaryTestDirectory}/tools/neo4j-community-4.4.0/conf/neo4j.conf" +touch "${temporaryTestDirectory}/tools/neo4j-community-4.4.0/conf/neo4j.conf.original.backup" + +# Use a template that contains dbms.* keys +cat > "${temporaryTestDirectory}/scripts/configuration/template-neo4j.conf" <<'EOF' +# v4 template +dbms.directories.import=/v4/import +dbms.tx_log.rotation.retention_policy=10 days +# End of configuration from template-neo4j.conf +EOF + +configureNeo4jExpectingSuccessUnderTest + +if ! grep -q "dbms.directories.import=" "${temporaryTestDirectory}/tools/neo4j-community-4.4.0/conf/neo4j.conf"; then + fail "${test_case_number}.) Test failed: Expected 'dbms.directories.import' entry not found in neo4j.conf for v4." +fi + +successful diff --git a/scripts/testDetectChangedFiles.sh b/scripts/testDetectChangedFiles.sh index 3dfea4b03..d8772a185 100755 --- a/scripts/testDetectChangedFiles.sh +++ b/scripts/testDetectChangedFiles.sh @@ -24,6 +24,12 @@ successful() { echo -e "testDetectChangedFiles: ${COLOR_SUCCESSFUL}Tests finished successfully.${COLOR_DEFAULT}" tearDown + # If sourced, return to caller; if executed directly, exit. + if [ "${BASH_SOURCE[0]}" != "$0" ]; then + return 0 + else + exit 0 + fi } fail() { @@ -128,5 +134,4 @@ if [ "${changeDetectionReturnCode}" = "0" ]; then fail "13.) Tests failed: Expected return code 0 if nothing changed with a protocol prefixed file, but got ${changeDetectionReturnCode}." fi -successful -return 0 \ No newline at end of file +successful \ No newline at end of file diff --git a/scripts/testDownloadMavenArtifacts.sh b/scripts/testDownloadMavenArtifacts.sh index c923c9ab2..76663865a 100755 --- a/scripts/testDownloadMavenArtifacts.sh +++ b/scripts/testDownloadMavenArtifacts.sh @@ -27,6 +27,13 @@ successful() { echo "" echo -e "${COLOR_DE_EMPHASIZED}${SCRIPT_NAME}:${COLOR_DEFAULT} ${COLOR_SUCCESSFUL}✅ Tests finished successfully.${COLOR_DEFAULT}" tearDown + + # If sourced, return to caller; if executed directly, exit. + if [ "${BASH_SOURCE[0]}" != "$0" ]; then + return 0 + else + exit 0 + fi } info() { @@ -78,7 +85,7 @@ downloadMavenArtifactsExpectingFailureUnderTest() { info "Starting tests...." # Create testing resources -temporaryTestDirectory=$(mktemp -d 2>/dev/null || mktemp -d -t 'temporaryTestDirectory_${SCRIPT_NAME}') +temporaryTestDirectory=$(mktemp -d 2>/dev/null || mktemp -d -t "temporaryTestDirectory_${SCRIPT_NAME}") mkdir -p "${temporaryTestDirectory}/artifacts" # ------- Integration Test Case @@ -146,5 +153,4 @@ echo "" info "${test_case_number}.) Should fail when the artifact coordinate has a wrong format (dry-run)." downloadMavenArtifactsExpectingFailureUnderTest "--dry-run" "org.apache.commons:commons-lang3-3.12.0" -successful -return 0 \ No newline at end of file +successful \ No newline at end of file diff --git a/scripts/testFilenameReferences.sh b/scripts/testFilenameReferences.sh index dc05eb068..5993e80f0 100755 --- a/scripts/testFilenameReferences.sh +++ b/scripts/testFilenameReferences.sh @@ -28,6 +28,12 @@ tearDown() { successful() { echo -e "${COLOR_DE_EMPHASIZED}${SCRIPT_NAME}:${COLOR_DEFAULT} ${COLOR_SUCCESSFUL}✅ Tests finished successfully.${COLOR_DEFAULT}" tearDown + # If sourced, return to caller; if executed directly, exit. + if [ "${BASH_SOURCE[0]}" != "$0" ]; then + return 0 + else + exit 0 + fi } info() { @@ -140,4 +146,3 @@ info "${test_case_number}.) Scan all .sh files for '/...*.yaml' references and v find_missing_file_references "yaml" successful -return 0 diff --git a/scripts/testInstallJavaScriptDependencies.sh b/scripts/testInstallJavaScriptDependencies.sh index 2f30d449d..b0fb0c047 100755 --- a/scripts/testInstallJavaScriptDependencies.sh +++ b/scripts/testInstallJavaScriptDependencies.sh @@ -27,6 +27,13 @@ successful() { echo "" echo -e "${COLOR_DE_EMPHASIZED}${SCRIPT_NAME}:${COLOR_DEFAULT} ${COLOR_SUCCESSFUL}✅ Tests finished successfully.${COLOR_DEFAULT}" tearDown + + # If sourced, return to caller; if executed directly, exit. + if [ "${BASH_SOURCE[0]}" != "$0" ]; then + return 0 + else + exit 0 + fi } info() { @@ -160,5 +167,4 @@ if [[ ! "${result}" == *"Installing JavaScript dependencies with yarn in"* ]]; t fail "❌ Test failed: Expected yarn installation attempt, but none was found:\n${result}" fi -successful -return 0 \ No newline at end of file +successful \ No newline at end of file diff --git a/scripts/useNeo4jHighMemoryProfile.sh b/scripts/useNeo4jHighMemoryProfile.sh new file mode 100755 index 000000000..f7e5f5e9c --- /dev/null +++ b/scripts/useNeo4jHighMemoryProfile.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash + +# Use the high memory profile and apply its configuration template on the local +# Neo4j Community Edition Graph Database (https://neo4j.com/download-center/#community). +# +# Prerequisites: +# - Run within the analysis workspace directory +# - Neo4j needs to be installed +# +# Example Usage: +# - ./../../scripts/useNeo4jHighMemoryProfile.sh +# +# Requires configureNeo4j.sh + +# Fail on any error ("-e" = exit on first error, "-o pipefail" exist on errors within piped commands) +set -o errexit -o pipefail + +## Get this "scripts" directory if not already set +# Even if $BASH_SOURCE is made for Bourne-like shells it is also supported by others and therefore here the preferred solution. +# CDPATH reduces the scope of the cd command to potentially prevent unintended directory changes. +# This way non-standard tools like readlink aren't needed. +SCRIPTS_DIR=${SCRIPTS_DIR:-$( CDPATH=. cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P )} # Repository directory containing the shell scripts + +# shellcheck disable=SC2034 +NEO4J_CONFIG_TEMPLATE=template-neo4j-high-memory.conf +. "${SCRIPTS_DIR}/configureNeo4j.sh" \ No newline at end of file