Merge pull request #535 from JohT/feature/modularize-neo4j-configuration-and-simplify-memory-increases

Modularize Neo4j configuration to support profile changes

Author: JohT

COMMANDS.md

@@ -19,6 +19,7 @@
 - [Validate Links in Markdown](#validate-links-in-markdown)
 - [Manual Setup](#manual-setup)
     - [Setup Neo4j Graph Database](#setup-neo4j-graph-database)
+    - [Change Neo4j configuration template](#change-neo4j-configuration-template)
     - [Start Neo4j Graph Database](#start-neo4j-graph-database)
     - [Setup jQAssistant Java Code Analyzer](#setup-jqassistant-java-code-analyzer)
     - [Download Maven Artifacts to analyze](#download-maven-artifacts-to-analyze)
@@ -51,13 +52,16 @@
 
 ## Start an Analysis
 
-An analysis is started with the script [analyze.sh](./scripts/analysis/analyze.sh).
+Before starting an analysis, setup your analysis as described in the [Getting Started](./GETTING_STARTED.md) guide.
+An analysis is then started with the script [analyze.sh](./scripts/analysis/analyze.sh).
 To run all analysis steps simple execute the following command:
 
 ```shell
 ./../../scripts/analysis/analyze.sh
 ```
 
+**Hint:** Within the analysis workspace directory you can simply run `analyze.sh` directly without the `../../` prefix since the script is also available in the analysis workspace.
+
 👉 See [scripts/examples/analyzeAxonFramework.sh](./scripts/examples/analyzeAxonFramework.sh) as an example script that combines all the above steps for a Java Project.
 👉 See [scripts/examples/analyzeReactRouter.sh](./scripts/examples/analyzeReactRouter.sh) as an example script that combines all the above steps for a Typescript Project.  
 👉 See [Code Structure Analysis Pipeline](./.github/workflows/internal-java-code-analysis.yml) on how to do this within a GitHub Actions Workflow.
@@ -183,11 +187,23 @@ If any of the script are not allowed to be executed use `chmod +x ./scripts/` fo
 Use [setupNeo4j.sh](./scripts/setupNeo4j.sh) to download [Neo4j](https://neo4j.com/download-center) and install the plugins [APOC](https://neo4j.com/labs/apoc/4.4) and [Graph Data Science](https://neo4j.com/product/graph-data-science).
 This script requires the environment variable NEO4J_INITIAL_PASSWORD to be set. It sets the initial password with a temporary `NEO4J_HOME` environment variable to not interfere with a possibly globally installed Neo4j installation.
 
+### Change Neo4j configuration template
+
+Use [configureNeo4j.sh](./scripts/configureNeo4j.sh) to apply a different Neo4j configuration template from the [scripts/configuration](./scripts/configuration/) directory. This can be useful to optimize Neo4j for different workloads. Example:
+
+```shell
+NEO4J_CONFIG_TEMPLATE=template-neo4j-high-memory.conf ./scripts/configureNeo4j.sh
+```
+
+**Hint:** In case you want to switch to the high memory profile as in the example, there is a simpler solution. Just run `useNeo4jHighMemoryProfile.sh` from the analysis workspace directory which will set the environment variable `NEO4J_CONFIG_TEMPLATE` and run `configureNeo4j.sh` for you.
+
 ### Start Neo4j Graph Database
 
 Use [startNeo4j.sh](./scripts/startNeo4j.sh) to start the locally installed [Neo4j](https://neo4j.com/download-center) Graph database.
 It runs the script with a temporary `NEO4J_HOME` environment variable to not interfere with a possibly globally installed Neo4j installation.
 
+**Hint:** Within the analysis workspace directory you can simply run `startNeo4j.sh` directly without the `../../` prefix since the script is also available in the analysis workspace.
+
 ### Setup jQAssistant Java Code Analyzer
 
 Use [setupJQAssistant.sh](./scripts/setupJQAssistant.sh) to download [jQAssistant](https://jqassistant.github.io/jqassistant/current).
@@ -346,6 +362,8 @@ execute_cypher ./cypher/Get_Graph_Data_Science_Library_Version.cypher a=1
 
 Use [stopNeo4j.sh](./scripts/stopNeo4j.sh) to stop the locally running Neo4j Graph Database. It does nothing if the database is already stopped. It runs the script with a temporary `NEO4J_HOME` environment variable to not interfere with a possibly globally installed Neo4j installation.
 
+**Hint:** Within the analysis workspace directory you can run `stopNeo4j.sh` directly without the `../../` prefix since the script is also directly available in the analysis workspace.
+
 ## Jupyter Notebook
 
 ### Create a report with executeJupyterNotebookReport.sh

README.md

@@ -249,7 +249,7 @@ The [Code Structure Analysis Pipeline](./.github/workflows/internal-java-code-an
   ENABLE_JUPYTER_NOTEBOOK_PDF_GENERATION=true ./../../scripts/analysis/analyze.sh
   ```
 
-- How can i disable git log data import?  
+- How can I disable git log data import?  
   👉 Set environment variable `IMPORT_GIT_LOG_DATA_IF_SOURCE_IS_PRESENT` to `none`. Example:  
 
   ```shell
@@ -272,34 +272,41 @@ The [Code Structure Analysis Pipeline](./.github/workflows/internal-java-code-an
   👉 The custom Jupyter Notebook metadata property `code_graph_analysis_pipeline_data_validation` can be set to choose a query from [cypher/Validation](./cypher/Validation) that will be executed preliminary to the notebook. If the query leads to at least one result, the validation succeeds and the notebook will be run. If the query leads to no result, the notebook will be skipped.
   For more details see [Data Availability Validation](./COMMANDS.md#data-availability-validation).
 
-- How can i increase the heap memory when scanning large Typescript projects?  
+- How can I increase the heap memory when scanning large Typescript projects?  
   👉 Use the environment variable TYPESCRIPT_SCAN_HEAP_MEMORY in megabyte (default = 4096):
 
   ```shell
   TYPESCRIPT_SCAN_HEAP_MEMORY=16384 ./../../scripts/analysis/analyze.sh
   ```
 
-- How can i continue on errors when scanning Typescript projects instead of cancelling the whole analysis?  
+- How can I continue on errors when scanning Typescript projects instead of cancelling the whole analysis?  
   👉 Use the profile `Neo4j-latest-continue-on-scan-errors` (default = `Neo4j-latest`):
 
   ```shell
   ./../../scripts/analysis/analyze.sh --profile Neo4j-latest-continue-on-scan-errors
   ```
 
-- How can i reduce the memory (RAM) consumption?  
+- How can I reduce the memory (RAM) consumption?  
   👉 Use the profile `Neo4j-latest-low-memory` (default = `Neo4j-latest`):
 
   ```shell
   ./../../scripts/analysis/analyze.sh --profile Neo4j-latest-low-memory
   ```
 
-- How can i increase the memory (RAM) consumption?  
+- How can I increase the memory (RAM) consumption?  
   👉 Use the profile `Neo4j-latest-high-memory` (default = `Neo4j-latest`):
 
   ```shell
   ./../../scripts/analysis/analyze.sh --profile Neo4j-latest-high-memory
   ```
 
+- How can i increase the memory (RAM) consumption afterwards, when the setup is already done?  
+  👉 Simply run `useNeo4jHighMemoryProfile.sh` in your analysis working directory, or:
+
+  ```shell
+  ./../../scripts/useNeo4jHighMemoryProfile.sh
+  ```
+
 ## 🕸 Web References
 
 - [code-graph-analysis-examples](https://github.com/JohT/code-graph-analysis-examples)

init.sh

@@ -79,6 +79,7 @@ mkdir -p "./${SOURCE_DIRECTORY}"
 createForwardingScript "./../../scripts/analysis/analyze.sh"
 createForwardingScript "./../../scripts/startNeo4j.sh"
 createForwardingScript "./../../scripts/stopNeo4j.sh"
+createForwardingScript "./../../scripts/useNeo4jHighMemoryProfile.sh"
 
 source "${SCRIPTS_DIR}/scripts/checkCompatibility.sh"
 

scripts/configuration/template-neo4j-high-memory.conf

@@ -1,6 +1,3 @@
-
-# The following static configuration entries were taken from "template-neo4j.conf".
-
 # Anonymous usage data reporting
 dbms.usage_report.enabled=false
 

scripts/configuration/template-neo4j-low-memory.conf

@@ -1,6 +1,3 @@
-
-# The following static configuration entries were taken from "template-neo4j.conf".
-
 # Anonymous usage data reporting
 dbms.usage_report.enabled=false
 

scripts/configuration/template-neo4j-v4-low-memory.conf

@@ -1,6 +1,3 @@
-
-# The following static configuration entries were taken from "template-neo4j.conf".
-
 # List of procedures and user defined functions that are allowed
 # full access to the database through unsupported/insecure internal APIs.
 dbms.security.procedures.unrestricted=apoc.*,gds.*

scripts/configuration/template-neo4j-v4.conf

@@ -1,6 +1,3 @@
-
-# The following static configuration entries were taken from "template-neo4j.conf".
-
 # List of procedures and user defined functions that are allowed
 # full access to the database through unsupported/insecure internal APIs.
 dbms.security.procedures.unrestricted=apoc.*,gds.*

scripts/configuration/template-neo4j.conf

@@ -1,6 +1,3 @@
-
-# The following static configuration entries were taken from "template-neo4j.conf".
-
 # Anonymous usage data reporting
 dbms.usage_report.enabled=false