diff --git a/CHANGELOG.md b/CHANGELOG.md index a3f287365..24381b015 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -264,7 +264,7 @@ For all details see: https://github.com/JohT/code-graph-analysis-pipeline/releas ### πŸ“– Jupyter Notebook Reports -* [External Dependencies](https://github.com/JohT/code-graph-analysis-examples/blob/main/analysis-results/AxonFramework/latest/external-dependencies-java/ExternalDependenciesJava.md) contains detailed information about external library usage ([Notebook](./jupyter/ExternalDependenciesJava.ipynb)). +* [External Dependencies](https://github.com/JohT/code-graph-analysis-examples/blob/main/analysis-results/AxonFramework/latest/external-dependencies-java/ExternalDependenciesJava.md) contains detailed information about external library usage ([Notebook](./domains/external-dependencies/explore/ExternalDependenciesJava.ipynb)). * [Object Oriented Design Quality Metrics](https://github.com/JohT/code-graph-analysis-examples/blob/main/analysis-results/AxonFramework/latest/object-oriented-design-metrics-java/ObjectOrientedDesignMetricsJava.md) is based on [OO Design Quality Metrics by Robert Martin](https://api.semanticscholar.org/CorpusID:18246616) ([Notebook](./jupyter/ObjectOrientedDesignMetricsJava.ipynb)). * [Overview](https://github.com/JohT/code-graph-analysis-examples/blob/main/analysis-results/AxonFramework/latest/overview-java/OverviewJava.md) contains overall statistics and details about methods and their complexity. ([Notebook](./jupyter/OverviewJava.ipynb)). * [Internal Dependencies](https://github.com/JohT/code-graph-analysis-examples/blob/main/analysis-results/AxonFramework/latest/internal-dependencies-java/InternalDependenciesJava.md) is based on [Analyze java package metrics in a graph database](https://joht.github.io/johtizen/data/2023/04/21/java-package-metrics-analysis.html) and also includes cyclic dependencies ([Notebook](./jupyter/InternalDependenciesJava.ipynb)). @@ -281,7 +281,7 @@ Here are some reports that utilize Neo4j's [Graph Data Science Library](https:// ### πŸ“– Other Reports -* [External Dependencies (CSV)](./scripts/reports/ExternalDependenciesCsv.sh) ([Example](https://github.com/JohT/code-graph-analysis-examples/blob/main/analysis-results/AxonFramework/latest/external-dependencies-csv/External_package_usage_overall.csv)) +* [External Dependencies (CSV)](./domains/external-dependencies/externalDependenciesCsv.sh) ([Example](https://github.com/JohT/code-graph-analysis-examples/blob/main/analysis-results/AxonFramework/latest/external-dependencies-csv/External_package_usage_overall.csv)) * [Object Oriented Design Metrics (CSV)](./scripts/reports/ObjectOrientedDesignMetricsCsv.sh) ([Example](https://github.com/JohT/code-graph-analysis-examples/blob/main/analysis-results/AxonFramework/latest/object-oriented-design-metrics-csv/MainSequenceAbstractnessInstabilityDistanceJava.csv)) * [Overview (CSV)](./scripts/reports/OverviewCsv.sh) ([Example](https://github.com/JohT/code-graph-analysis-examples/blob/main/analysis-results/AxonFramework/latest/overview-csv/Cyclomatic_Method_Complexity.csv)) * [Internal Dependencies - Cyclic (CSV)](./scripts/reports/InternalDependenciesCsv.sh) ([Example](https://github.com/JohT/code-graph-analysis-examples/blob/main/analysis-results/AxonFramework/latest/internal-dependencies-csv/Cyclic_Dependencies_Breakdown_Backward_Only.csv)) diff --git a/README.md b/README.md index 412de9472..0390eeb7a 100644 --- a/README.md +++ b/README.md @@ -46,7 +46,7 @@ Curious? Explore the examples at [code-graph-analysis-examples](https://github.c Here is an overview of [Jupyter Notebooks](https://jupyter.org) reports from [code-graph-analysis-examples](https://github.com/JohT/code-graph-analysis-examples). For a complete list, see the [Jupyter Notebook Report Reference](#page_with_curl-jupyter-notebook-report-reference). -- [External Dependencies](https://github.com/JohT/code-graph-analysis-examples/blob/main/analysis-results/AxonFramework/latest/external-dependencies-java/ExternalDependenciesJava.md) contains detailed information about external library usage ([Notebook](./jupyter/ExternalDependenciesJava.ipynb)). +- [External Dependencies](https://github.com/JohT/code-graph-analysis-examples/blob/main/analysis-results/AxonFramework/latest/external-dependencies-java/ExternalDependenciesJava.md) contains detailed information about external library usage ([Notebook](./domains/external-dependencies/explore/ExternalDependenciesJava.ipynb)). - [Git History](https://github.com/JohT/code-graph-analysis-examples/blob/main/analysis-results/AxonFramework/latest/git-history-general/GitHistoryGeneral.md) contains information about the git history of the analyzed code ([Notebook](./jupyter/GitHistoryGeneral.ipynb)). - [Internal Dependencies](https://github.com/JohT/code-graph-analysis-examples/blob/main/analysis-results/AxonFramework/latest/internal-dependencies-java/InternalDependenciesJava.md) is based on [Analyze java package metrics in a graph database](https://joht.github.io/johtizen/data/2023/04/21/java-package-metrics-analysis.html) and also includes cyclic dependencies ([Notebook](./jupyter/InternalDependenciesJava.ipynb)). - [Method Metrics](https://github.com/JohT/code-graph-analysis-examples/blob/main/analysis-results/AxonFramework/latest/method-metrics-java/MethodMetricsJava.md) shows how the effective number of lines of code and the cyclomatic complexity are distributed across the methods in the code ([Notebook](./jupyter/MethodMetricsJava.ipynb)). diff --git a/cypher/External_Dependencies/Label_external_types_and_annotations.cypher b/cypher/Java/Label_external_types_and_annotations.cypher similarity index 100% rename from cypher/External_Dependencies/Label_external_types_and_annotations.cypher rename to cypher/Java/Label_external_types_and_annotations.cypher diff --git a/cypher/External_Dependencies/Remove_external_type_and_annotation_labels.cypher b/cypher/Java/Remove_external_type_and_annotation_labels.cypher similarity index 100% rename from cypher/External_Dependencies/Remove_external_type_and_annotation_labels.cypher rename to cypher/Java/Remove_external_type_and_annotation_labels.cypher diff --git a/domains/external-dependencies/README.md b/domains/external-dependencies/README.md new file mode 100644 index 000000000..e13b6506c --- /dev/null +++ b/domains/external-dependencies/README.md @@ -0,0 +1,79 @@ +# External Dependencies Domain + +This directory contains the implementation and resources for analysing external dependencies within the Code Graph Analysis Pipeline. It follows the vertical-slice domain pattern: all Cypher queries, Python chart scripts, and report templates needed for this analysis live here. + +## Entry Points + +The following scripts are discovered and invoked automatically by the central compilation scripts in [scripts/reports/compilations/](../../scripts/reports/compilations/). They are found by filename pattern. + +- [externalDependenciesCsv.sh](./externalDependenciesCsv.sh): Entry point for CSV reports based on Cypher queries. Discovered by `CsvReports.sh` (`*Csv.sh` pattern). +- [externalDependenciesPython.sh](./externalDependenciesPython.sh): Entry point for Python-based chart generation. Discovered by `PythonReports.sh` (`*Python.sh` pattern). +- [externalDependenciesMarkdown.sh](./externalDependenciesMarkdown.sh): Entry point for the Markdown summary report. Discovered by `MarkdownReports.sh` (`*Markdown.sh` pattern). + +## Folder Structure + +- [explore](./explore/): Original Jupyter notebooks for interactive, exploratory analysis. Marked with `code_graph_analysis_pipeline_data_validation: ValidateAlwaysFalse` so they are not automatically executed by the pipeline. +- [queries](./queries/): All Cypher queries for identifying and quantifying external dependencies. These are self-contained copies from [cypher/External_Dependencies/](../../cypher/External_Dependencies/). +- [summary](./summary/): Markdown template and assembly script for the summary report. + +## Prerequisites + +This domain requires the following to be in place before running. These are provided by the central pipeline ([scripts/prepareAnalysis.sh](../../scripts/prepareAnalysis.sh)) and are **not** set up by this domain. + +### Graph database + +- Neo4j must be running and accessible at `bolt://localhost:7687`. +- `NEO4J_INITIAL_PASSWORD` environment variable must be set. +- Artifacts must already have been scanned and loaded by jQAssistant, creating `DEPENDS_ON` relationships between `Type` nodes. + +### Node labels (Java) + +The following labels must exist on `Type` nodes before external dependency analysis can run. They are created by Cypher queries in [cypher/Types/](../../cypher/Types/): + +- `PrimitiveType` β€” primitive types like `int`, `boolean` +- `Void` β€” void return type +- `JavaType` β€” built-in Java standard library types (e.g. `java.lang.*`, `java.util.*`) +- `ResolvedDuplicateType` β€” deduplicated types that appear in multiple jars + +Without these labels, `Label_external_types_and_annotations.cypher` cannot correctly distinguish external types from internal and built-in ones. + +### Relationship weight properties (Java) + +The following properties must exist on `DEPENDS_ON` relationships between `Package` nodes. They are set by Cypher queries in [cypher/DependsOn_Relationship_Weights/](../../cypher/DependsOn_Relationship_Weights/): + +- `weight` β€” sum of type-level dependency weights between two packages +- `weightInterfaces` β€” subset of `weight` attributable to interface dependencies + +### TypeScript enrichment + +For TypeScript projects, the following must be completed by [cypher/Typescript_Enrichment/](../../cypher/Typescript_Enrichment/): + +- Module properties set on `Module` nodes: `namespace`, `moduleName`, `isNodeModule`, `isExternalImport` +- `IS_IMPLEMENTED_IN` relationships linking `ExternalModule` nodes to their resolved internal `Module` nodes +- `DEPENDS_ON` relationships propagated to resolved modules +- NPM packages linked to their corresponding `ExternalModule` nodes via `PROVIDED_BY_NPM_DEPENDENCY` + +### General enrichment + +- `name` and `extension` properties on `File` nodes β€” set by [cypher/General_Enrichment/](../../cypher/General_Enrichment/). + +## What This Domain Produces + +### CSV reports (`reports/external-dependencies/`) + +One CSV file per Cypher query covering: + +- **Java packages**: overall usage, spread, per-artifact, per-type, distribution, aggregated, Maven POM declared dependencies +- **TypeScript modules and namespaces**: overall usage, spread, per-internal-module, aggregated +- **Package management**: `package.json` dependency occurrence and combinations + +### SVG charts (`reports/external-dependencies/`) + +Python-generated charts from [externalDependencyCharts.py](./externalDependencyCharts.py): + +- **Java**: pie charts for most-used and most-spread packages (by types and by packages, with drill-down into "others"), stacked bar charts for per-artifact breakdown, scatter plots for aggregated usage patterns +- **TypeScript**: pie charts for modules and namespaces (usage and spread, with drill-down) + +### Markdown summary (`reports/external-dependencies/external_dependencies_report.md`) + +A structured report designed to be readable by both humans and AI agents. Includes key findings, tables, chart references, and architectural recommendations such as Hexagonal Architecture patterns and Anti-Corruption Layer candidates. diff --git a/jupyter/ExternalDependenciesJava.ipynb b/domains/external-dependencies/explore/ExternalDependenciesJava.ipynb similarity index 99% rename from jupyter/ExternalDependenciesJava.ipynb rename to domains/external-dependencies/explore/ExternalDependenciesJava.ipynb index 0c533f790..7df0545bd 100644 --- a/jupyter/ExternalDependenciesJava.ipynb +++ b/domains/external-dependencies/explore/ExternalDependenciesJava.ipynb @@ -1726,7 +1726,7 @@ } ], "celltoolbar": "Tags", - "code_graph_analysis_pipeline_data_validation": "ValidateJavaExternalDependencies", + "code_graph_analysis_pipeline_data_validation": "ValidateAlwaysFalse", "kernelspec": { "display_name": "codegraph", "language": "python", diff --git a/jupyter/ExternalDependenciesTypescript.ipynb b/domains/external-dependencies/explore/ExternalDependenciesTypescript.ipynb similarity index 99% rename from jupyter/ExternalDependenciesTypescript.ipynb rename to domains/external-dependencies/explore/ExternalDependenciesTypescript.ipynb index c8de0f004..5059b823a 100644 --- a/jupyter/ExternalDependenciesTypescript.ipynb +++ b/domains/external-dependencies/explore/ExternalDependenciesTypescript.ipynb @@ -1629,7 +1629,7 @@ } ], "celltoolbar": "Tags", - "code_graph_analysis_pipeline_data_validation": "ValidateTypescriptModuleDependencies", + "code_graph_analysis_pipeline_data_validation": "ValidateAlwaysFalse", "kernelspec": { "display_name": "codegraph", "language": "python", diff --git a/domains/external-dependencies/externalDependenciesCsv.sh b/domains/external-dependencies/externalDependenciesCsv.sh new file mode 100755 index 000000000..bdfa1d170 --- /dev/null +++ b/domains/external-dependencies/externalDependenciesCsv.sh @@ -0,0 +1,161 @@ +#!/usr/bin/env bash + +# Executes Cypher queries to generate external dependency CSV reports. +# It requires an already running Neo4j graph database with already scanned and analyzed artifacts. +# The results will be written into the sub directory reports/external-dependencies. +# Dynamically triggered by "CsvReports.sh". + +# Note that "scripts/prepareAnalysis.sh" is required to run prior to this script. +# Specifically, type labels (ExternalType, ExternalAnnotation) and weight properties on DEPENDS_ON +# relationships must already exist. See README.md for the full prerequisites list. + +# Requires executeQueryFunctions.sh, cleanupAfterReportGeneration.sh + +# Fail on any error ("-e" = exit on first error, "-o pipefail" exist on errors within piped commands) +set -o errexit -o pipefail + +# Overrideable Constants (defaults also defined in sub scripts) +REPORTS_DIRECTORY=${REPORTS_DIRECTORY:-"reports"} + +## Get this "domains/external-dependencies" directory if not already set +# Even if $BASH_SOURCE is made for Bourne-like shells it is also supported by others and therefore here the preferred solution. +# CDPATH reduces the scope of the cd command to potentially prevent unintended directory changes. +# This way non-standard tools like readlink aren't needed. +EXTERNAL_DEPENDENCIES_SCRIPT_DIR=${EXTERNAL_DEPENDENCIES_SCRIPT_DIR:-$(CDPATH=. cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)} +echo "externalDependenciesCsv: EXTERNAL_DEPENDENCIES_SCRIPT_DIR=${EXTERNAL_DEPENDENCIES_SCRIPT_DIR}" + +# Get the "scripts" directory by navigating two levels up from this domain directory. +SCRIPTS_DIR=${SCRIPTS_DIR:-"${EXTERNAL_DEPENDENCIES_SCRIPT_DIR}/../../scripts"} + +# Cypher query directory within this domain +EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR=${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR:-"${EXTERNAL_DEPENDENCIES_SCRIPT_DIR}/queries"} + +# Define functions to execute a cypher query from within a given file like "execute_cypher" and "execute_cypher_queries_until_results" +source "${SCRIPTS_DIR}/executeQueryFunctions.sh" + +# Create report directory +REPORT_NAME="external-dependencies" +FULL_REPORT_DIRECTORY="${REPORTS_DIRECTORY}/${REPORT_NAME}" +mkdir -p "${FULL_REPORT_DIRECTORY}" + +echo "externalDependenciesCsv: $(date +'%Y-%m-%dT%H:%M:%S%z') Starting external dependencies CSV reports..." + +# Ensure ExternalType and ExternalAnnotation labels exist on Java types. +# If they already exist, the labelling query is skipped. +execute_cypher_queries_until_results \ + "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/List_external_Java_types_used.cypher" \ + "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/Label_external_types_and_annotations.cypher" > /dev/null + +# -- Java Package Reports -------------------------------------------------- + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_package_usage_overall.cypher" \ + > "${FULL_REPORT_DIRECTORY}/External_package_usage_overall.csv" + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_package_usage_spread.cypher" \ + > "${FULL_REPORT_DIRECTORY}/External_package_usage_spread.csv" + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_second_level_package_usage_overall.cypher" \ + > "${FULL_REPORT_DIRECTORY}/External_second_level_package_usage_overall.csv" + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_second_level_package_usage_spread.cypher" \ + > "${FULL_REPORT_DIRECTORY}/External_second_level_package_usage_spread.csv" + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_package_usage_per_type.cypher" \ + > "${FULL_REPORT_DIRECTORY}/External_package_usage_per_type.csv" + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_package_usage_per_artifact.cypher" \ + > "${FULL_REPORT_DIRECTORY}/External_package_usage_per_artifact.csv" + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_package_usage_per_artifact_sorted.cypher" \ + > "${FULL_REPORT_DIRECTORY}/External_package_usage_per_artifact_sorted.csv" + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_package_usage_per_artifact_sorted_top.cypher" \ + > "${FULL_REPORT_DIRECTORY}/External_package_usage_per_artifact_sorted_top.csv" + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_package_usage_per_artifact_distribution.cypher" \ + > "${FULL_REPORT_DIRECTORY}/External_package_usage_per_artifact_distribution.csv" + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_package_usage_per_artifact_package_aggregated.cypher" \ + > "${FULL_REPORT_DIRECTORY}/External_package_usage_per_artifact_package_aggregated.csv" + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_package_usage_per_artifact_and_package.cypher" \ + > "${FULL_REPORT_DIRECTORY}/External_package_usage_per_artifact_and_package.csv" + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_package_usage_per_artifact_and_external_package.cypher" \ + > "${FULL_REPORT_DIRECTORY}/External_package_usage_per_artifact_and_external_package.csv" + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_second_level_package_usage_per_artifact_and_external_package.cypher" \ + > "${FULL_REPORT_DIRECTORY}/External_second_level_package_usage_per_artifact_and_external_package.csv" + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_package_usage_per_internal_package_count.cypher" \ + > "${FULL_REPORT_DIRECTORY}/External_package_usage_per_internal_package_count.csv" + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_package_usage_per_artifact_and_package_with_annotations.cypher" \ + > "${FULL_REPORT_DIRECTORY}/External_package_usage_per_artifact_and_package_with_annotations.csv" + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_package_usage_per_type_distribution_with_annotations.cypher" \ + > "${FULL_REPORT_DIRECTORY}/External_package_usage_per_type_distribution_with_annotations.csv" + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_package_levels.cypher" \ + > "${FULL_REPORT_DIRECTORY}/External_package_levels.csv" + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_package_name_elements.cypher" \ + > "${FULL_REPORT_DIRECTORY}/External_package_name_elements.csv" + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_types_per_artifact_using_requires.cypher" \ + > "${FULL_REPORT_DIRECTORY}/External_types_per_artifact_using_requires.csv" + +# -- Maven POM Reports ----------------------------------------------------- + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/Maven_POMs_and_their_declared_dependencies.cypher" \ + > "${FULL_REPORT_DIRECTORY}/Maven_POM_dependencies.csv" + +# -- TypeScript Module Reports --------------------------------------------- + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_module_usage_overall_for_Typescript.cypher" \ + > "${FULL_REPORT_DIRECTORY}/External_module_usage_overall_for_Typescript.csv" + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_module_usage_spread_for_Typescript.cypher" \ + > "${FULL_REPORT_DIRECTORY}/External_module_usage_spread_for_Typescript.csv" + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_namespace_usage_overall_for_Typescript.cypher" \ + > "${FULL_REPORT_DIRECTORY}/External_namespace_usage_overall_for_Typescript.csv" + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_namespace_usage_spread_for_Typescript.cypher" \ + > "${FULL_REPORT_DIRECTORY}/External_namespace_usage_spread_for_Typescript.csv" + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_module_usage_per_internal_module_sorted_for_Typescript.cypher" \ + > "${FULL_REPORT_DIRECTORY}/External_module_usage_per_internal_module_sorted_for_Typescript.csv" + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_namespace_usage_per_internal_module_sorted_for_Typescript.cypher" \ + > "${FULL_REPORT_DIRECTORY}/External_namespace_usage_per_internal_module_sorted_for_Typescript.csv" + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_module_usage_per_internal_module_aggregated_for_Typescript.cypher" \ + > "${FULL_REPORT_DIRECTORY}/External_module_usage_per_internal_module_aggregated_for_Typescript.csv" + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_module_usage_per_internal_module_distribution_for_Typescript.cypher" \ + > "${FULL_REPORT_DIRECTORY}/External_module_usage_per_internal_module_distribution_for_Typescript.csv" + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/List_external_modules_resolved_to_internal_ones_for_Typescript.cypher" \ + > "${FULL_REPORT_DIRECTORY}/External_modules_resolved_to_internal_ones_for_Typescript.csv" + +# -- Package Management Reports -------------------------------------------- + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/Package_json_dependencies_occurrence.cypher" \ + > "${FULL_REPORT_DIRECTORY}/Package_json_dependencies_occurrence.csv" + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/Package_json_dependencies_by_package.cypher" \ + > "${FULL_REPORT_DIRECTORY}/Package_json_dependencies_by_package.csv" + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/Package_json_dependencies_combinations.cypher" \ + > "${FULL_REPORT_DIRECTORY}/Package_json_dependencies_combinations.csv" + +execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/Package_json_dependencies_combinations_with_versions.cypher" \ + > "${FULL_REPORT_DIRECTORY}/Package_json_dependencies_combinations_with_versions.csv" + +# -------------------------------------------------------------------------- + +# Clean up: delete empty CSV files that were generated when no data was available +source "${SCRIPTS_DIR}/cleanupAfterReportGeneration.sh" "${FULL_REPORT_DIRECTORY}" + +echo "externalDependenciesCsv: $(date +'%Y-%m-%dT%H:%M:%S%z') Successfully finished." diff --git a/domains/external-dependencies/externalDependenciesMarkdown.sh b/domains/external-dependencies/externalDependenciesMarkdown.sh new file mode 100755 index 000000000..00195fc5d --- /dev/null +++ b/domains/external-dependencies/externalDependenciesMarkdown.sh @@ -0,0 +1,28 @@ +#!/usr/bin/env bash + +# This script is dynamically triggered by "MarkdownReports.sh" when report "All" or "Markdown" are enabled. +# It is designed as an entry point and delegates the execution to the dedicated "externalDependenciesSummary.sh" script that does the "heavy lifting". + +# Note that "scripts/prepareAnalysis.sh" is required to run prior to this script. +# Note that either "externalDependenciesCsv.sh" or "externalDependenciesPython.sh" is required to run prior to this script. + +# Requires externalDependenciesSummary.sh + +# Fail on any error ("-e" = exit on first error, "-o pipefail" exist on errors within piped commands) +set -o errexit -o pipefail + +# Overrideable Constants (defaults also defined in sub scripts) +REPORTS_DIRECTORY=${REPORTS_DIRECTORY:-"reports"} + +## Get this "domains/external-dependencies" directory if not already set +# Even if $BASH_SOURCE is made for Bourne-like shells it is also supported by others and therefore here the preferred solution. +# CDPATH reduces the scope of the cd command to potentially prevent unintended directory changes. +# This way non-standard tools like readlink aren't needed. +EXTERNAL_DEPENDENCIES_SCRIPT_DIR=${EXTERNAL_DEPENDENCIES_SCRIPT_DIR:-$(CDPATH=. cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)} +# echo "externalDependenciesMarkdown: EXTERNAL_DEPENDENCIES_SCRIPT_DIR=${EXTERNAL_DEPENDENCIES_SCRIPT_DIR}" + +# Get the "summary" directory by taking the path of this script and selecting "summary". +EXTERNAL_DEPENDENCIES_SUMMARY_DIR=${EXTERNAL_DEPENDENCIES_SUMMARY_DIR:-"${EXTERNAL_DEPENDENCIES_SCRIPT_DIR}/summary"} # Contains everything (scripts, queries, templates) to create the Markdown summary report for external dependencies + +# Delegate the execution to the responsible script. +source "${EXTERNAL_DEPENDENCIES_SUMMARY_DIR}/externalDependenciesSummary.sh" diff --git a/domains/external-dependencies/externalDependenciesPython.sh b/domains/external-dependencies/externalDependenciesPython.sh new file mode 100755 index 000000000..533297ebb --- /dev/null +++ b/domains/external-dependencies/externalDependenciesPython.sh @@ -0,0 +1,84 @@ +#!/usr/bin/env bash + +# Generates external dependency charts as SVG files using Python. +# It requires an already running Neo4j graph database with already scanned and analyzed artifacts. +# The results will be written into the sub directory reports/external-dependencies. +# Dynamically triggered by "PythonReports.sh". + +# Note that "scripts/prepareAnalysis.sh" is required to run prior to this script. +# Note that "externalDependenciesCsv.sh" is required to run prior to this script +# so that ExternalType and ExternalAnnotation labels exist in the graph. + +# Requires executeQueryFunctions.sh, cleanupAfterReportGeneration.sh + +# Fail on any error ("-e" = exit on first error, "-o pipefail" exist on errors within piped commands) +set -o errexit -o pipefail + +# Overrideable Constants (defaults also defined in sub scripts) +REPORTS_DIRECTORY=${REPORTS_DIRECTORY:-"reports"} + +## Get this "domains/external-dependencies" directory if not already set +# Even if $BASH_SOURCE is made for Bourne-like shells it is also supported by others and therefore here the preferred solution. +# CDPATH reduces the scope of the cd command to potentially prevent unintended directory changes. +# This way non-standard tools like readlink aren't needed. +EXTERNAL_DEPENDENCIES_SCRIPT_DIR=${EXTERNAL_DEPENDENCIES_SCRIPT_DIR:-$(CDPATH=. cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)} +echo "externalDependenciesPython: EXTERNAL_DEPENDENCIES_SCRIPT_DIR=${EXTERNAL_DEPENDENCIES_SCRIPT_DIR}" + +# Get the "scripts" directory by navigating two levels up from this domain directory. +SCRIPTS_DIR=${SCRIPTS_DIR:-"${EXTERNAL_DEPENDENCIES_SCRIPT_DIR}/../../scripts"} + +# Function to display script usage +usage() { + echo -e "${COLOR_ERROR}" >&2 + echo "Usage: $0 [--verbose]" >&2 + echo -e "${COLOR_DEFAULT}" >&2 + exit 1 +} + +# Default values +verboseMode="" # either "" or "--verbose" + +# Parse command line arguments +while [[ $# -gt 0 ]]; do + key="$1" + + case ${key} in + --verbose) + verboseMode="--verbose" + ;; + *) + echo -e "${COLOR_ERROR}externalDependenciesPython: Error: Unknown option: ${key}${COLOR_DEFAULT}" >&2 + usage + ;; + esac + shift || true # ignore error when there are no more arguments +done + +# Define functions to execute a cypher query from within a given file like "execute_cypher" +source "${SCRIPTS_DIR}/executeQueryFunctions.sh" + +# Create report directory +REPORT_NAME="external-dependencies" +FULL_REPORT_DIRECTORY="${REPORTS_DIRECTORY}/${REPORT_NAME}" +mkdir -p "${FULL_REPORT_DIRECTORY}" + +echo "externalDependenciesPython: $(date +'%Y-%m-%dT%H:%M:%S%z') Starting external dependencies Python chart generation..." + +# -- Java Charts ----------------------------------------------------------- +echo "externalDependenciesPython: $(date +'%Y-%m-%dT%H:%M:%S%z') Generating Java charts..." +time python "${EXTERNAL_DEPENDENCIES_SCRIPT_DIR}/externalDependencyCharts.py" \ + --language Java \ + --report_directory "${FULL_REPORT_DIRECTORY}" \ + ${verboseMode} + +# -- TypeScript Charts ----------------------------------------------------- +echo "externalDependenciesPython: $(date +'%Y-%m-%dT%H:%M:%S%z') Generating TypeScript charts..." +time python "${EXTERNAL_DEPENDENCIES_SCRIPT_DIR}/externalDependencyCharts.py" \ + --language Typescript \ + --report_directory "${FULL_REPORT_DIRECTORY}" \ + ${verboseMode} + +# Clean-up after report generation. Empty reports will be deleted. +source "${SCRIPTS_DIR}/cleanupAfterReportGeneration.sh" "${FULL_REPORT_DIRECTORY}" + +echo "externalDependenciesPython: $(date +'%Y-%m-%dT%H:%M:%S%z') Successfully finished." diff --git a/domains/external-dependencies/externalDependencyCharts.py b/domains/external-dependencies/externalDependencyCharts.py new file mode 100644 index 000000000..3d3f4a3fa --- /dev/null +++ b/domains/external-dependencies/externalDependencyCharts.py @@ -0,0 +1,760 @@ +#!/usr/bin/env python + +# Generates external dependency charts as SVG files from Neo4j graph data. +# Charts are saved to the report directory and referenced by the Markdown summary report. +# +# Input Parameters: +# --language Java|Typescript (required) +# --report_directory path to write SVG files +# --verbose optional finer-grained logging +# +# Prerequisites: +# - Neo4j must be running with scanned artifacts loaded. +# - ExternalType and ExternalAnnotation labels must already exist (see externalDependenciesCsv.sh). +# - NEO4J_INITIAL_PASSWORD environment variable must be set. + +import os +import sys +import argparse +import typing +from typing import LiteralString, cast + +import pandas as pd + +import matplotlib +matplotlib.use('Agg') # Non-interactive backend β€” required for headless script execution +import matplotlib.pyplot as plot + +from neo4j import GraphDatabase, Driver + +SCRIPT_NAME = "externalDependencyCharts" +JAVA_LANGUAGE = "Java" +TYPESCRIPT_LANGUAGE = "Typescript" +DRILLDOWN_SECONDARY_THRESHOLD_PERCENT = 0.3 + + +class Parameters: + def __init__( + self, + language: str, + queries_directory: str, + report_directory: str, + verbose: bool, + ) -> None: + self.language = language + self.queries_directory = queries_directory + self.report_directory = report_directory + self.verbose = verbose + + def __repr__(self) -> str: + return ( + f"Parameters(language={self.language!r}, " + f"queries_directory={self.queries_directory!r}, " + f"report_directory={self.report_directory!r}, " + f"verbose={self.verbose})" + ) + + @staticmethod + def log_dependency_versions() -> None: + print("---------------------------------------") + print(f"Python version: {sys.version}") + from pandas import __version__ as pandas_version + print(f"pandas version: {pandas_version}") + from matplotlib import __version__ as matplotlib_version + print(f"matplotlib version: {matplotlib_version}") + from neo4j import __version__ as neo4j_version + print(f"neo4j version: {neo4j_version}") + print("---------------------------------------") + + +def parse_parameters() -> Parameters: + script_directory = os.path.dirname(os.path.abspath(__file__)) + default_queries_directory = os.path.join(script_directory, "queries") + + parser = argparse.ArgumentParser( + description="Generates external dependency charts as SVG files from Neo4j graph data." + ) + parser.add_argument( + "--language", + type=str, + required=True, + choices=[JAVA_LANGUAGE, TYPESCRIPT_LANGUAGE], + help="Programming language to analyse: Java or Typescript", + ) + parser.add_argument( + "--report_directory", + type=str, + default="", + help="Path to the directory where SVG files are written", + ) + parser.add_argument( + "--verbose", + action="store_true", + default=False, + help="Enable verbose mode for detailed logging", + ) + args = parser.parse_args() + + parameters = Parameters( + language=args.language, + queries_directory=default_queries_directory, + report_directory=args.report_directory, + verbose=args.verbose, + ) + if parameters.verbose: + print(parameters) + Parameters.log_dependency_versions() + return parameters + + +def connect_to_graph_database() -> Driver: + password = os.environ.get("NEO4J_INITIAL_PASSWORD") + if not password: + print( + f"{SCRIPT_NAME}: Environment variable NEO4J_INITIAL_PASSWORD must be set and non-empty " + "before connecting to Neo4j.", + file=sys.stderr, + ) + sys.exit(1) + + graph_driver = GraphDatabase.driver( + uri="bolt://localhost:7687", + auth=("neo4j", password), + ) + graph_driver.verify_connectivity() + print(f"{SCRIPT_NAME}: Successfully connected to Neo4j") + return graph_driver + + +def load_query_results(cypher_file_path: str, verbose: bool, driver: Driver) -> pd.DataFrame: + """Reads a .cypher file and executes it against the graph database, returning a DataFrame.""" + with open(cypher_file_path, "r", encoding="utf-8") as cypher_file: + query = cypher_file.read() + if verbose: + print(f"{SCRIPT_NAME}: Querying {os.path.basename(cypher_file_path)}") + records, summary, keys = driver.execute_query(cast(LiteralString, query)) + if verbose: + print(f"{SCRIPT_NAME}: Got {len(records)} rows after {summary.result_available_after} ms") + return pd.DataFrame([record.values() for record in records], columns=keys) + + +def chart_file_path(name: str, report_directory: str, verbose: bool) -> str: + """Returns the full SVG file path for a named chart.""" + path = os.path.join(report_directory, name.replace(" ", "_") + ".svg") + if verbose: + print(f"{SCRIPT_NAME}: Saving {path}") + return path + + +# ── Pure data transformation functions ──────────────────────────────────────── + + +def add_percentage_column(data_frame: pd.DataFrame, value_column: str) -> pd.DataFrame: + """Returns a copy of the DataFrame with a percentage column added (based on column total).""" + result = data_frame.copy() + percent_column = value_column + "Percent" + result[percent_column] = result[value_column] / result[value_column].sum() * 100.0 + return result + + +def group_small_values_into_others( + data_frame: pd.DataFrame, + value_column: str, + name_column: str, + threshold_percent: float, +) -> pd.DataFrame: + """ + Groups rows whose percentage share of the total falls below threshold_percent + into a single 'others' row. Returns index=name, columns=[value, percent], sorted descending. + """ + result = data_frame[[name_column, value_column]].copy() + percent_column = value_column + "Percent" + result[percent_column] = result[value_column] / result[value_column].sum() * 100.0 + result[name_column] = result[name_column].astype(str) + result.loc[result[percent_column] < threshold_percent, name_column] = "others" + result = result.groupby(name_column).sum() + return result.sort_values(by=percent_column, ascending=False) + + +def filter_entries_below_percentage_threshold( + data_frame: pd.DataFrame, + value_column: str, + threshold_percent: float, +) -> pd.DataFrame: + """ + Returns only rows whose percentage share of the *original* total is strictly + below threshold_percent. Used to drill down into the 'others' slice. + """ + result = add_percentage_column(data_frame, value_column) + percent_column = value_column + "Percent" + result = result[result[percent_column] < threshold_percent] + return result.reset_index(drop=True) + + +def explode_pie_slice( + grouped_data: pd.DataFrame, + slice_name: str = "others", + base_offset: float = 0.02, + emphasized_offset: float = 0.2, +) -> typing.List[float]: + """ + Returns per-slice explode offsets for matplotlib pie charts. + The slice matching slice_name is emphasized with a larger offset. + """ + return [ + (emphasized_offset if index == slice_name else base_offset) + for index in grouped_data.index + ] + + +def select_top_artifacts_by_package_count( + per_artifact_data: pd.DataFrame, + top_count: int = 15, +) -> typing.List[str]: + """Returns the names of the top artifacts ranked by total internal packages using external dependencies.""" + totals = ( + per_artifact_data + .groupby("artifactName")["numberOfPackages"] + .sum() + .nlargest(top_count) + ) + return totals.index.tolist() + + +def build_external_package_per_artifact_pivot( + per_artifact_data: pd.DataFrame, + package_name_column: str, + top_artifacts: typing.List[str], +) -> pd.DataFrame: + """ + Creates a pivot table: rows = external packages, columns = artifacts, + values = number of internal packages. Limited to the given top artifacts. + """ + filtered = per_artifact_data[per_artifact_data["artifactName"].isin(top_artifacts)] + return filtered.pivot_table( + values="numberOfPackages", + index=package_name_column, + columns="artifactName", + fill_value=0, + aggfunc="sum", + ) + + +def derive_second_level_package_name(full_package_name: str) -> str: + """Returns the first two dot-separated segments of a package name (e.g. 'javax.xml' from 'javax.xml.stream').""" + parts = full_package_name.split(".") + return ".".join(parts[:2]) if len(parts) >= 2 else full_package_name + + +# ── Chart rendering functions ────────────────────────────────────────────────── + + +def save_pie_chart(grouped_data: pd.DataFrame, title: str, file_path: str) -> None: + """ + Renders a pie chart from a grouped DataFrame (index=name, columns=[value, percent]) and saves as SVG. + Skips silently when the input is empty. + """ + if grouped_data.empty: + print(f"{SCRIPT_NAME}: No data for '{title}', skipping chart.") + return + + value_column = grouped_data.columns[0] + percent_column = grouped_data.columns[1] if len(grouped_data.columns) > 1 else value_column + "Percent" + total = grouped_data[value_column].sum() + + def format_slice_label(percentage: float) -> str: + return f"{percentage:.2f}% ({total * percentage / 100.0:.0f})" + + offsets = explode_pie_slice(grouped_data) + fig, ax = plot.subplots(figsize=(9, 9)) + grouped_data.plot( + kind="pie", + y=percent_column, + ylabel="", + legend=True, + labeldistance=None, + autopct=format_slice_label, + textprops={"fontsize": 6}, + pctdistance=1.15, + cmap="nipy_spectral", + ax=ax, + explode=offsets, + ) + plot.title(title, pad=15) + ax.legend(bbox_to_anchor=(1.08, 1), loc="upper left") + plot.savefig(file_path, bbox_inches="tight") + plot.close(fig) + + +def save_stacked_bar_chart( + pivot_data: pd.DataFrame, + title: str, + xlabel: str, + ylabel: str, + file_path: str, +) -> None: + """ + Renders a stacked bar chart (transposed pivot: artifacts on x-axis, external packages stacked). + Skips silently when the input is empty. + """ + if pivot_data.empty: + print(f"{SCRIPT_NAME}: No data for '{title}', skipping chart.") + return + + fig, ax = plot.subplots(figsize=(14, 8)) + pivot_data.transpose().plot( + kind="bar", + grid=True, + title=title, + xlabel=xlabel, + ylabel=ylabel, + stacked=True, + legend=True, + cmap="nipy_spectral", + ax=ax, + ) + ax.legend(bbox_to_anchor=(1.0, 1.0)) + plot.tight_layout() + plot.savefig(file_path, bbox_inches="tight") + plot.close(fig) + + +def annotate_artifact_scatter_point( + data_frame: pd.DataFrame, + x_column: str, + y_column: str, + sort_by_highest: typing.List[str], +) -> None: + """ + Annotates one artifact in the current scatter plot with an arrow and label. + The artifact is selected by sorting the DataFrame: columns in sort_by_highest descend, others ascend. + """ + sort_columns = [x_column, y_column, "artifactPackages", "artifactName"] + ascending = [column not in sort_by_highest for column in sort_columns] + row = data_frame.sort_values(by=sort_columns, ascending=ascending).iloc[0] + + label_box = dict(boxstyle="round4,pad=0.5", fc="w", alpha=0.8) + plot.annotate( + row["artifactName"], + xy=(row[x_column], row[y_column]), + xycoords="data", + xytext=(-30, -15), + textcoords="offset points", + size=6, + bbox=label_box, + arrowprops=dict(arrowstyle="-|>", mutation_scale=10, color="black"), + ) + + +def save_scatter_chart( + data_frame: pd.DataFrame, + x_column: str, + y_column: str, + size_column: str, + color_column: str, + title: str, + xlabel: str, + ylabel: str, + file_path: str, + annotation_sort_configs: typing.List[typing.List[str]], +) -> None: + """ + Renders a scatter plot and annotates several notable artifacts. + Each list in annotation_sort_configs defines which columns to sort descending + for selecting one annotation point. + Skips silently when the input is empty. + """ + if data_frame.empty: + print(f"{SCRIPT_NAME}: No data for '{title}', skipping chart.") + return + + fig, ax = plot.subplots(figsize=(10, 7)) + data_frame.plot( + kind="scatter", + title=title, + x=x_column, + y=y_column, + s=size_column, + c=color_column, + xlabel=xlabel, + ylabel=ylabel, + cmap="nipy_spectral", + ax=ax, + ) + for sort_config in annotation_sort_configs: + if not data_frame.empty: + annotate_artifact_scatter_point(data_frame, x_column, y_column, sort_by_highest=sort_config) + + plot.savefig(file_path, bbox_inches="tight") + plot.close(fig) + + +# ── Reusable pie chart pair generator ───────────────────────────────────────── + + +def save_pie_chart_pair( + source_data: pd.DataFrame, + value_column: str, + name_column: str, + chart_name_prefix: str, + primary_threshold_percent: float, + report_directory: str, + verbose: bool, +) -> None: + """ + Generates two pie charts from source_data: + 1. Entries at or above the threshold grouped by name (small ones merged into 'others'). + 2. Drill-down of the 'others' group: only entries below the threshold. + """ + grouped = group_small_values_into_others(source_data, value_column, name_column, primary_threshold_percent) + save_pie_chart( + grouped_data=grouped, + title=f"{chart_name_prefix} (β‰₯{primary_threshold_percent}%)", + file_path=chart_file_path(f"{chart_name_prefix}_above_threshold", report_directory, verbose), + ) + + others_entries = filter_entries_below_percentage_threshold(source_data, value_column, primary_threshold_percent) + drilldown_grouped = group_small_values_into_others( + others_entries, value_column, name_column, DRILLDOWN_SECONDARY_THRESHOLD_PERCENT + ) + save_pie_chart( + grouped_data=drilldown_grouped, + title=f"{chart_name_prefix} β€” others drill-down (<{primary_threshold_percent}%)", + file_path=chart_file_path(f"{chart_name_prefix}_others_drilldown", report_directory, verbose), + ) + + +# ── Java chart generation ────────────────────────────────────────────────────── + + +def generate_java_charts(queries_directory: str, report_directory: str, verbose: bool, driver: Driver) -> None: + """Generates all external dependency charts for Java packages.""" + print(f"{SCRIPT_NAME}: Generating Java external dependency charts...") + + overall_data = load_query_results( + os.path.join(queries_directory, "External_package_usage_overall.cypher"), verbose, driver + ) + second_level_overall_data = load_query_results( + os.path.join(queries_directory, "External_second_level_package_usage_overall.cypher"), verbose, driver + ) + spread_data = load_query_results( + os.path.join(queries_directory, "External_package_usage_spread.cypher"), verbose, driver + ) + second_level_spread_data = load_query_results( + os.path.join(queries_directory, "External_second_level_package_usage_spread.cypher"), verbose, driver + ) + per_artifact_and_package_data = load_query_results( + os.path.join(queries_directory, "External_package_usage_per_artifact_and_external_package.cypher"), verbose, driver + ) + aggregated_data = load_query_results( + os.path.join(queries_directory, "External_package_usage_per_artifact_package_aggregated.cypher"), verbose, driver + ) + + # ── Top external packages (Table 1 equivalent) ──────────────────────────── + if not overall_data.empty: + top20 = overall_data.head(20) + save_pie_chart_pair( + source_data=top20, + value_column="numberOfExternalCallerTypes", + name_column="externalPackageName", + chart_name_prefix="Java_Top_external_packages_by_types", + primary_threshold_percent=0.7, + report_directory=report_directory, + verbose=verbose, + ) + save_pie_chart_pair( + source_data=top20, + value_column="numberOfExternalCallerPackages", + name_column="externalPackageName", + chart_name_prefix="Java_Top_external_packages_by_packages", + primary_threshold_percent=0.7, + report_directory=report_directory, + verbose=verbose, + ) + + # ── Second-level package grouping (Table 2 equivalent) ──────────────────── + if not second_level_overall_data.empty: + top20_second_level = second_level_overall_data.head(20) + save_pie_chart_pair( + source_data=top20_second_level, + value_column="numberOfExternalCallerTypes", + name_column="externalSecondLevelPackageName", + chart_name_prefix="Java_Top_second_level_packages_by_types", + primary_threshold_percent=0.7, + report_directory=report_directory, + verbose=verbose, + ) + save_pie_chart_pair( + source_data=top20_second_level, + value_column="numberOfExternalCallerPackages", + name_column="externalSecondLevelPackageName", + chart_name_prefix="Java_Top_second_level_packages_by_packages", + primary_threshold_percent=0.7, + report_directory=report_directory, + verbose=verbose, + ) + + # ── Most spread external packages (Table 3 equivalent) ──────────────────── + if not spread_data.empty: + top20_spread = spread_data.head(20) + save_pie_chart_pair( + source_data=top20_spread, + value_column="sumNumberOfTypes", + name_column="externalPackageName", + chart_name_prefix="Java_Most_spread_packages_by_types", + primary_threshold_percent=0.5, + report_directory=report_directory, + verbose=verbose, + ) + save_pie_chart_pair( + source_data=top20_spread, + value_column="sumNumberOfPackages", + name_column="externalPackageName", + chart_name_prefix="Java_Most_spread_packages_by_packages", + primary_threshold_percent=0.5, + report_directory=report_directory, + verbose=verbose, + ) + + # ── Most spread second-level packages (Table 4 equivalent) ──────────────── + if not second_level_spread_data.empty: + top20_second_level_spread = second_level_spread_data.head(20) + save_pie_chart_pair( + source_data=top20_second_level_spread, + value_column="sumNumberOfTypes", + name_column="externalSecondLevelPackageName", + chart_name_prefix="Java_Most_spread_second_level_packages_by_types", + primary_threshold_percent=0.5, + report_directory=report_directory, + verbose=verbose, + ) + save_pie_chart_pair( + source_data=top20_second_level_spread, + value_column="sumNumberOfPackages", + name_column="externalSecondLevelPackageName", + chart_name_prefix="Java_Most_spread_second_level_packages_by_packages", + primary_threshold_percent=0.5, + report_directory=report_directory, + verbose=verbose, + ) + + # ── Stacked bar: external packages per artifact (Table 7 equivalent) ────── + if not per_artifact_and_package_data.empty: + top_artifacts = select_top_artifacts_by_package_count(per_artifact_and_package_data) + + full_package_pivot = build_external_package_per_artifact_pivot( + per_artifact_and_package_data, "externalPackageName", top_artifacts + ) + save_stacked_bar_chart( + pivot_data=full_package_pivot, + title="External package usage per artifact (top 15)", + xlabel="artifact", + ylabel="number of internal packages", + file_path=chart_file_path( + "Java_External_package_usage_per_artifact_stacked", report_directory, verbose + ), + ) + + per_artifact_second_level = per_artifact_and_package_data.copy() + per_artifact_second_level["externalSecondLevelPackageName"] = ( + per_artifact_second_level["externalPackageName"] + .apply(derive_second_level_package_name) + ) + second_level_pivot = build_external_package_per_artifact_pivot( + per_artifact_second_level, "externalSecondLevelPackageName", top_artifacts + ) + save_stacked_bar_chart( + pivot_data=second_level_pivot, + title="External package usage per artifact β€” second-level grouping (top 15)", + xlabel="artifact", + ylabel="number of internal packages", + file_path=chart_file_path( + "Java_External_second_level_package_usage_per_artifact_stacked", report_directory, verbose + ), + ) + + # ── Scatter: aggregated package usage patterns (Table 13 equivalent) ────── + if not aggregated_data.empty: + save_scatter_chart( + data_frame=aggregated_data, + x_column="numberOfExternalPackages", + y_column="maxNumberOfPackagesPercentage", + size_column="artifactPackages", + color_column="stdNumberOfPackagesPercentage", + title="External package usage β€” max internal packages %", + xlabel="external package count", + ylabel="max percentage of internal packages", + file_path=chart_file_path( + "Java_External_package_usage_max_internal_packages_percent", report_directory, verbose + ), + annotation_sort_configs=[ + ["numberOfExternalPackages", "maxNumberOfPackagesPercentage"], + ["maxNumberOfPackagesPercentage"], + [], + ], + ) + save_scatter_chart( + data_frame=aggregated_data, + x_column="numberOfExternalPackages", + y_column="medNumberOfPackagesPercentage", + size_column="artifactPackages", + color_column="stdNumberOfPackagesPercentage", + title="External package usage β€” median internal packages %", + xlabel="external package count", + ylabel="median percentage of internal packages", + file_path=chart_file_path( + "Java_External_package_usage_median_internal_packages_percent", report_directory, verbose + ), + annotation_sort_configs=[], + ) + + print(f"{SCRIPT_NAME}: Java charts complete.") + + +# ── TypeScript chart generation ──────────────────────────────────────────────── + + +def generate_typescript_charts(queries_directory: str, report_directory: str, verbose: bool, driver: Driver) -> None: + """Generates all external dependency charts for TypeScript modules and namespaces.""" + print(f"{SCRIPT_NAME}: Generating TypeScript external dependency charts...") + + module_overall_data = load_query_results( + os.path.join(queries_directory, "External_module_usage_overall_for_Typescript.cypher"), verbose, driver + ) + namespace_overall_data = load_query_results( + os.path.join(queries_directory, "External_namespace_usage_overall_for_Typescript.cypher"), verbose, driver + ) + module_spread_data = load_query_results( + os.path.join(queries_directory, "External_module_usage_spread_for_Typescript.cypher"), verbose, driver + ) + namespace_spread_data = load_query_results( + os.path.join(queries_directory, "External_namespace_usage_spread_for_Typescript.cypher"), verbose, driver + ) + + # ── Module usage overall ─────────────────────────────────────────────────── + if not module_overall_data.empty: + top20_modules = module_overall_data.head(20) + save_pie_chart_pair( + source_data=top20_modules, + value_column="numberOfExternalCallerElements", + name_column="externalModuleName", + chart_name_prefix="Typescript_Top_external_modules_by_elements", + primary_threshold_percent=0.7, + report_directory=report_directory, + verbose=verbose, + ) + save_pie_chart_pair( + source_data=top20_modules, + value_column="numberOfExternalCallerModules", + name_column="externalModuleName", + chart_name_prefix="Typescript_Top_external_modules_by_modules", + primary_threshold_percent=0.7, + report_directory=report_directory, + verbose=verbose, + ) + + # ── Namespace usage overall ──────────────────────────────────────────────── + if not namespace_overall_data.empty: + top20_namespaces = namespace_overall_data.head(20) + save_pie_chart_pair( + source_data=top20_namespaces, + value_column="numberOfExternalCallerElements", + name_column="externalNamespaceName", + chart_name_prefix="Typescript_Top_external_namespaces_by_elements", + primary_threshold_percent=0.7, + report_directory=report_directory, + verbose=verbose, + ) + save_pie_chart_pair( + source_data=top20_namespaces, + value_column="numberOfExternalCallerModules", + name_column="externalNamespaceName", + chart_name_prefix="Typescript_Top_external_namespaces_by_modules", + primary_threshold_percent=0.7, + report_directory=report_directory, + verbose=verbose, + ) + + # ── Module spread ────────────────────────────────────────────────────────── + if not module_spread_data.empty: + top20_module_spread = module_spread_data.head(20) + save_pie_chart_pair( + source_data=top20_module_spread, + value_column="sumNumberOfUsedExternalDeclarations", + name_column="externalModuleName", + chart_name_prefix="Typescript_Most_spread_modules_by_declarations", + primary_threshold_percent=0.5, + report_directory=report_directory, + verbose=verbose, + ) + save_pie_chart_pair( + source_data=top20_module_spread, + value_column="numberOfInternalModules", + name_column="externalModuleName", + chart_name_prefix="Typescript_Most_spread_modules_by_modules", + primary_threshold_percent=0.5, + report_directory=report_directory, + verbose=verbose, + ) + + # ── Namespace spread ─────────────────────────────────────────────────────── + if not namespace_spread_data.empty: + top20_namespace_spread = namespace_spread_data.head(20) + save_pie_chart_pair( + source_data=top20_namespace_spread, + value_column="sumNumberOfUsedExternalDeclarations", + name_column="externalModuleNamespace", + chart_name_prefix="Typescript_Most_spread_namespaces_by_declarations", + primary_threshold_percent=0.5, + report_directory=report_directory, + verbose=verbose, + ) + save_pie_chart_pair( + source_data=top20_namespace_spread, + value_column="numberOfInternalModules", + name_column="externalModuleNamespace", + chart_name_prefix="Typescript_Most_spread_namespaces_by_modules", + primary_threshold_percent=0.5, + report_directory=report_directory, + verbose=verbose, + ) + + print(f"{SCRIPT_NAME}: TypeScript charts complete.") + + +# ── Entry point ──────────────────────────────────────────────────────────────── + + +def main() -> None: + parameters = parse_parameters() + + if parameters.report_directory: + os.makedirs(parameters.report_directory, exist_ok=True) + + driver = connect_to_graph_database() + + if parameters.language == JAVA_LANGUAGE: + generate_java_charts( + queries_directory=parameters.queries_directory, + report_directory=parameters.report_directory, + verbose=parameters.verbose, + driver=driver, + ) + elif parameters.language == TYPESCRIPT_LANGUAGE: + generate_typescript_charts( + queries_directory=parameters.queries_directory, + report_directory=parameters.report_directory, + verbose=parameters.verbose, + driver=driver, + ) + else: + print(f"{SCRIPT_NAME}: Error: Unknown language '{parameters.language}'", file=sys.stderr) + driver.close() + sys.exit(1) + + driver.close() + print(f"{SCRIPT_NAME}: Neo4j connection closed.") + + +if __name__ == "__main__": + main() diff --git a/cypher/External_Dependencies/External_module_usage_overall_for_Typescript.cypher b/domains/external-dependencies/queries/External_module_usage_overall_for_Typescript.cypher similarity index 100% rename from cypher/External_Dependencies/External_module_usage_overall_for_Typescript.cypher rename to domains/external-dependencies/queries/External_module_usage_overall_for_Typescript.cypher diff --git a/cypher/External_Dependencies/External_module_usage_per_internal_module_aggregated_for_Typescript.cypher b/domains/external-dependencies/queries/External_module_usage_per_internal_module_aggregated_for_Typescript.cypher similarity index 100% rename from cypher/External_Dependencies/External_module_usage_per_internal_module_aggregated_for_Typescript.cypher rename to domains/external-dependencies/queries/External_module_usage_per_internal_module_aggregated_for_Typescript.cypher diff --git a/cypher/External_Dependencies/External_module_usage_per_internal_module_distribution_for_Typescript.cypher b/domains/external-dependencies/queries/External_module_usage_per_internal_module_distribution_for_Typescript.cypher similarity index 100% rename from cypher/External_Dependencies/External_module_usage_per_internal_module_distribution_for_Typescript.cypher rename to domains/external-dependencies/queries/External_module_usage_per_internal_module_distribution_for_Typescript.cypher diff --git a/cypher/External_Dependencies/External_module_usage_per_internal_module_sorted_for_Typescript.cypher b/domains/external-dependencies/queries/External_module_usage_per_internal_module_sorted_for_Typescript.cypher similarity index 100% rename from cypher/External_Dependencies/External_module_usage_per_internal_module_sorted_for_Typescript.cypher rename to domains/external-dependencies/queries/External_module_usage_per_internal_module_sorted_for_Typescript.cypher diff --git a/cypher/External_Dependencies/External_module_usage_spread_for_Typescript.cypher b/domains/external-dependencies/queries/External_module_usage_spread_for_Typescript.cypher similarity index 98% rename from cypher/External_Dependencies/External_module_usage_spread_for_Typescript.cypher rename to domains/external-dependencies/queries/External_module_usage_spread_for_Typescript.cypher index 6a0fcc826..ccccd87ee 100644 --- a/cypher/External_Dependencies/External_module_usage_spread_for_Typescript.cypher +++ b/domains/external-dependencies/queries/External_module_usage_spread_for_Typescript.cypher @@ -23,7 +23,7 @@ UNWIND internalElementList AS internalElement ,COLLECT(DISTINCT externalDeclaration.globalFqn )[0..9] AS externalDeclarationsExamples ,count (DISTINCT internalElement.globalFqn) AS internalElementsCount ,COLLECT(DISTINCT internalElement.globalFqn )[0..9] AS internalElementsExamples - ,100.0 / internalModulesCountOverall + ,100.0 / internalElementsCountOverall * count(DISTINCT internalElement.globalFqn) AS internalElementsCallingExternalRate // Group by external module diff --git a/cypher/External_Dependencies/External_namespace_usage_overall_for_Typescript.cypher b/domains/external-dependencies/queries/External_namespace_usage_overall_for_Typescript.cypher similarity index 100% rename from cypher/External_Dependencies/External_namespace_usage_overall_for_Typescript.cypher rename to domains/external-dependencies/queries/External_namespace_usage_overall_for_Typescript.cypher diff --git a/cypher/External_Dependencies/External_namespace_usage_per_internal_module_sorted_for_Typescript.cypher b/domains/external-dependencies/queries/External_namespace_usage_per_internal_module_sorted_for_Typescript.cypher similarity index 100% rename from cypher/External_Dependencies/External_namespace_usage_per_internal_module_sorted_for_Typescript.cypher rename to domains/external-dependencies/queries/External_namespace_usage_per_internal_module_sorted_for_Typescript.cypher diff --git a/cypher/External_Dependencies/External_namespace_usage_spread_for_Typescript.cypher b/domains/external-dependencies/queries/External_namespace_usage_spread_for_Typescript.cypher similarity index 98% rename from cypher/External_Dependencies/External_namespace_usage_spread_for_Typescript.cypher rename to domains/external-dependencies/queries/External_namespace_usage_spread_for_Typescript.cypher index 42cb20823..dd2a25e6c 100644 --- a/cypher/External_Dependencies/External_namespace_usage_spread_for_Typescript.cypher +++ b/domains/external-dependencies/queries/External_namespace_usage_spread_for_Typescript.cypher @@ -23,7 +23,7 @@ UNWIND internalElementList AS internalElement ,COLLECT(DISTINCT externalDeclaration.globalFqn )[0..9] AS externalDeclarationsExamples ,count (DISTINCT internalElement.globalFqn) AS internalElementsCount ,COLLECT(DISTINCT internalElement.globalFqn )[0..9] AS internalElementsExamples - ,100.0 / internalModulesCountOverall + ,100.0 / internalElementsCountOverall * count(DISTINCT internalElement.globalFqn) AS internalElementsCallingExternalRate // Group by external module namespace diff --git a/cypher/External_Dependencies/External_package_levels.cypher b/domains/external-dependencies/queries/External_package_levels.cypher similarity index 100% rename from cypher/External_Dependencies/External_package_levels.cypher rename to domains/external-dependencies/queries/External_package_levels.cypher diff --git a/cypher/External_Dependencies/External_package_name_elements.cypher b/domains/external-dependencies/queries/External_package_name_elements.cypher similarity index 100% rename from cypher/External_Dependencies/External_package_name_elements.cypher rename to domains/external-dependencies/queries/External_package_name_elements.cypher diff --git a/cypher/External_Dependencies/External_package_usage_overall.cypher b/domains/external-dependencies/queries/External_package_usage_overall.cypher similarity index 95% rename from cypher/External_Dependencies/External_package_usage_overall.cypher rename to domains/external-dependencies/queries/External_package_usage_overall.cypher index b026375f3..8d0881b24 100644 --- a/cypher/External_Dependencies/External_package_usage_overall.cypher +++ b/domains/external-dependencies/queries/External_package_usage_overall.cypher @@ -15,7 +15,6 @@ UNWIND typeList AS type ,count(externalDependency) AS numberOfExternalTypeCalls ,sum(externalDependency.weight) AS numberOfExternalTypeCallsWeighted ,collect(DISTINCT externalType.name) AS externalTypeNames -where numberOfExternalTypeCalls <> numberOfExternalCallerTypes RETURN externalPackageName ,numberOfExternalCallerPackages ,numberOfExternalCallerTypes diff --git a/cypher/External_Dependencies/External_package_usage_per_artifact.cypher b/domains/external-dependencies/queries/External_package_usage_per_artifact.cypher similarity index 100% rename from cypher/External_Dependencies/External_package_usage_per_artifact.cypher rename to domains/external-dependencies/queries/External_package_usage_per_artifact.cypher diff --git a/cypher/External_Dependencies/External_package_usage_per_artifact_and_external_package.cypher b/domains/external-dependencies/queries/External_package_usage_per_artifact_and_external_package.cypher similarity index 100% rename from cypher/External_Dependencies/External_package_usage_per_artifact_and_external_package.cypher rename to domains/external-dependencies/queries/External_package_usage_per_artifact_and_external_package.cypher diff --git a/cypher/External_Dependencies/External_package_usage_per_artifact_and_package.cypher b/domains/external-dependencies/queries/External_package_usage_per_artifact_and_package.cypher similarity index 100% rename from cypher/External_Dependencies/External_package_usage_per_artifact_and_package.cypher rename to domains/external-dependencies/queries/External_package_usage_per_artifact_and_package.cypher diff --git a/cypher/External_Dependencies/External_package_usage_per_artifact_and_package_with_annotations.cypher b/domains/external-dependencies/queries/External_package_usage_per_artifact_and_package_with_annotations.cypher similarity index 100% rename from cypher/External_Dependencies/External_package_usage_per_artifact_and_package_with_annotations.cypher rename to domains/external-dependencies/queries/External_package_usage_per_artifact_and_package_with_annotations.cypher diff --git a/cypher/External_Dependencies/External_package_usage_per_artifact_distribution.cypher b/domains/external-dependencies/queries/External_package_usage_per_artifact_distribution.cypher similarity index 100% rename from cypher/External_Dependencies/External_package_usage_per_artifact_distribution.cypher rename to domains/external-dependencies/queries/External_package_usage_per_artifact_distribution.cypher diff --git a/cypher/External_Dependencies/External_package_usage_per_artifact_package_aggregated.cypher b/domains/external-dependencies/queries/External_package_usage_per_artifact_package_aggregated.cypher similarity index 98% rename from cypher/External_Dependencies/External_package_usage_per_artifact_package_aggregated.cypher rename to domains/external-dependencies/queries/External_package_usage_per_artifact_package_aggregated.cypher index 4e35cbebd..2c5a8958b 100644 --- a/cypher/External_Dependencies/External_package_usage_per_artifact_package_aggregated.cypher +++ b/domains/external-dependencies/queries/External_package_usage_per_artifact_package_aggregated.cypher @@ -13,7 +13,7 @@ UNWIND typeList AS type MATCH (type)-[:DEPENDS_ON]->(externalType:ExternalType) MATCH (typePackage:Package)-[:CONTAINS]->(type) -// Filter out dependencies to exxternal annotations +// Filter out dependencies to external annotations WHERE NOT externalType:ExternalAnnotation WITH artifactName ,leidenCommunityId diff --git a/cypher/External_Dependencies/External_package_usage_per_artifact_sorted.cypher b/domains/external-dependencies/queries/External_package_usage_per_artifact_sorted.cypher similarity index 100% rename from cypher/External_Dependencies/External_package_usage_per_artifact_sorted.cypher rename to domains/external-dependencies/queries/External_package_usage_per_artifact_sorted.cypher diff --git a/cypher/External_Dependencies/External_package_usage_per_artifact_sorted_top.cypher b/domains/external-dependencies/queries/External_package_usage_per_artifact_sorted_top.cypher similarity index 100% rename from cypher/External_Dependencies/External_package_usage_per_artifact_sorted_top.cypher rename to domains/external-dependencies/queries/External_package_usage_per_artifact_sorted_top.cypher diff --git a/cypher/External_Dependencies/External_package_usage_per_internal_package_count.cypher b/domains/external-dependencies/queries/External_package_usage_per_internal_package_count.cypher similarity index 100% rename from cypher/External_Dependencies/External_package_usage_per_internal_package_count.cypher rename to domains/external-dependencies/queries/External_package_usage_per_internal_package_count.cypher diff --git a/cypher/External_Dependencies/External_package_usage_per_type.cypher b/domains/external-dependencies/queries/External_package_usage_per_type.cypher similarity index 100% rename from cypher/External_Dependencies/External_package_usage_per_type.cypher rename to domains/external-dependencies/queries/External_package_usage_per_type.cypher diff --git a/cypher/External_Dependencies/External_package_usage_per_type_distribution_with_annotations.cypher b/domains/external-dependencies/queries/External_package_usage_per_type_distribution_with_annotations.cypher similarity index 100% rename from cypher/External_Dependencies/External_package_usage_per_type_distribution_with_annotations.cypher rename to domains/external-dependencies/queries/External_package_usage_per_type_distribution_with_annotations.cypher diff --git a/cypher/External_Dependencies/External_package_usage_spread.cypher b/domains/external-dependencies/queries/External_package_usage_spread.cypher similarity index 100% rename from cypher/External_Dependencies/External_package_usage_spread.cypher rename to domains/external-dependencies/queries/External_package_usage_spread.cypher diff --git a/cypher/External_Dependencies/External_second_level_package_usage_overall.cypher b/domains/external-dependencies/queries/External_second_level_package_usage_overall.cypher similarity index 95% rename from cypher/External_Dependencies/External_second_level_package_usage_overall.cypher rename to domains/external-dependencies/queries/External_second_level_package_usage_overall.cypher index 8f0c27314..1c8ad6815 100644 --- a/cypher/External_Dependencies/External_second_level_package_usage_overall.cypher +++ b/domains/external-dependencies/queries/External_second_level_package_usage_overall.cypher @@ -15,7 +15,6 @@ UNWIND typeList AS type ,count(externalDependency) AS numberOfExternalTypeCalls ,sum(externalDependency.weight) AS numberOfExternalTypeCallsWeighted ,collect(DISTINCT externalType.name) AS externalTypeNames -where numberOfExternalTypeCalls <> numberOfExternalCallerTypes RETURN externalSecondLevelPackageName ,numberOfExternalCallerPackages ,numberOfExternalCallerTypes diff --git a/cypher/External_Dependencies/External_second_level_package_usage_per_artifact_and_external_package.cypher b/domains/external-dependencies/queries/External_second_level_package_usage_per_artifact_and_external_package.cypher similarity index 100% rename from cypher/External_Dependencies/External_second_level_package_usage_per_artifact_and_external_package.cypher rename to domains/external-dependencies/queries/External_second_level_package_usage_per_artifact_and_external_package.cypher diff --git a/cypher/External_Dependencies/External_second_level_package_usage_spread.cypher b/domains/external-dependencies/queries/External_second_level_package_usage_spread.cypher similarity index 100% rename from cypher/External_Dependencies/External_second_level_package_usage_spread.cypher rename to domains/external-dependencies/queries/External_second_level_package_usage_spread.cypher diff --git a/cypher/External_Dependencies/External_types_per_artifact_using_requires.cypher b/domains/external-dependencies/queries/External_types_per_artifact_using_requires.cypher similarity index 100% rename from cypher/External_Dependencies/External_types_per_artifact_using_requires.cypher rename to domains/external-dependencies/queries/External_types_per_artifact_using_requires.cypher diff --git a/domains/external-dependencies/queries/Label_external_types_and_annotations.cypher b/domains/external-dependencies/queries/Label_external_types_and_annotations.cypher new file mode 100644 index 000000000..72ff76478 --- /dev/null +++ b/domains/external-dependencies/queries/Label_external_types_and_annotations.cypher @@ -0,0 +1,13 @@ +// Label external types and external annotations. Requires 'Label_base_java_types', 'Label_buildin_java_types' and 'Label_resolved_duplicate_types' of 'Types' directory. + + MATCH (type:Type&!PrimitiveType&!Void&!JavaType&!ResolvedDuplicateType&!TS) + WITH type + ,type.byteCodeVersion IS NULL AS isExternalType + ,exists((type)<-[:OF_TYPE]-()<-[:ANNOTATED_BY]-()) AS isAnnotation + WITH type + ,isExternalType + ,CASE WHEN isAnnotation THEN [1] ELSE [] END AS annotation + WHERE isExternalType +FOREACH (x in annotation | SET type:ExternalAnnotation) + SET type:ExternalType + RETURN labels(type), count(type) as numberOfExternalTypes \ No newline at end of file diff --git a/cypher/External_Dependencies/List_external_Java_types_used.cypher b/domains/external-dependencies/queries/List_external_Java_types_used.cypher similarity index 100% rename from cypher/External_Dependencies/List_external_Java_types_used.cypher rename to domains/external-dependencies/queries/List_external_Java_types_used.cypher diff --git a/cypher/External_Dependencies/List_external_modules_resolved_to_internal_ones_for_Typescript.cypher b/domains/external-dependencies/queries/List_external_modules_resolved_to_internal_ones_for_Typescript.cypher similarity index 78% rename from cypher/External_Dependencies/List_external_modules_resolved_to_internal_ones_for_Typescript.cypher rename to domains/external-dependencies/queries/List_external_modules_resolved_to_internal_ones_for_Typescript.cypher index 65e421aa5..ef26d8db5 100644 --- a/cypher/External_Dependencies/List_external_modules_resolved_to_internal_ones_for_Typescript.cypher +++ b/domains/external-dependencies/queries/List_external_modules_resolved_to_internal_ones_for_Typescript.cypher @@ -4,8 +4,8 @@ OPTIONAL MATCH (project:TS:Project)-[:CONTAINS]->(module) WITH project.name AS projectName ,count(DISTINCT module) AS resolvedModuleCount - ,COUNT { (modules:Module) } AS totalModuleCount - ,COUNT { (externalModules:ExternalModule) } AS totalExternalModuleCount + ,COUNT { (project)-[:CONTAINS]->(modules:TS:Module) } AS totalModuleCount + ,COUNT { (ext:TS:ExternalModule)-[:IS_IMPLEMENTED_IN]->(m:TS:Module)<-[:CONTAINS]-(project) } AS totalExternalModuleCount ,collect(DISTINCT module.fileName + ' -> ' + external.globalFqn)[0..4] AS resolvedExamples RETURN projectName ,resolvedModuleCount diff --git a/cypher/External_Dependencies/Maven_POMs_and_their_declared_dependencies.cypher b/domains/external-dependencies/queries/Maven_POMs_and_their_declared_dependencies.cypher similarity index 100% rename from cypher/External_Dependencies/Maven_POMs_and_their_declared_dependencies.cypher rename to domains/external-dependencies/queries/Maven_POMs_and_their_declared_dependencies.cypher diff --git a/cypher/External_Dependencies/Package_json_dependencies_by_package.cypher b/domains/external-dependencies/queries/Package_json_dependencies_by_package.cypher similarity index 100% rename from cypher/External_Dependencies/Package_json_dependencies_by_package.cypher rename to domains/external-dependencies/queries/Package_json_dependencies_by_package.cypher diff --git a/cypher/External_Dependencies/Package_json_dependencies_combinations.cypher b/domains/external-dependencies/queries/Package_json_dependencies_combinations.cypher similarity index 100% rename from cypher/External_Dependencies/Package_json_dependencies_combinations.cypher rename to domains/external-dependencies/queries/Package_json_dependencies_combinations.cypher diff --git a/cypher/External_Dependencies/Package_json_dependencies_combinations_with_versions.cypher b/domains/external-dependencies/queries/Package_json_dependencies_combinations_with_versions.cypher similarity index 100% rename from cypher/External_Dependencies/Package_json_dependencies_combinations_with_versions.cypher rename to domains/external-dependencies/queries/Package_json_dependencies_combinations_with_versions.cypher diff --git a/cypher/External_Dependencies/Package_json_dependencies_occurrence.cypher b/domains/external-dependencies/queries/Package_json_dependencies_occurrence.cypher similarity index 100% rename from cypher/External_Dependencies/Package_json_dependencies_occurrence.cypher rename to domains/external-dependencies/queries/Package_json_dependencies_occurrence.cypher diff --git a/domains/external-dependencies/queries/Remove_external_type_and_annotation_labels.cypher b/domains/external-dependencies/queries/Remove_external_type_and_annotation_labels.cypher new file mode 100644 index 000000000..b4c4e69ec --- /dev/null +++ b/domains/external-dependencies/queries/Remove_external_type_and_annotation_labels.cypher @@ -0,0 +1,4 @@ +// Remove external type and annotation labels + + MATCH (externalType:ExternalType) + REMOVE externalType:ExternalType:ExternalAnnotation \ No newline at end of file diff --git a/domains/external-dependencies/summary/externalDependenciesSummary.sh b/domains/external-dependencies/summary/externalDependenciesSummary.sh new file mode 100644 index 000000000..32dd91ad8 --- /dev/null +++ b/domains/external-dependencies/summary/externalDependenciesSummary.sh @@ -0,0 +1,205 @@ +#!/usr/bin/env bash + +# Creates a Markdown report summarising all external dependency analysis results. +# It requires an already running Neo4j graph database with already scanned and analyzed artifacts. +# The results will be written into the sub directory reports/external-dependencies. +# Dynamically triggered by "MarkdownReports.sh" via "externalDependenciesMarkdown.sh". + +# Note that "scripts/prepareAnalysis.sh" is required to run prior to this script. +# Note that either "externalDependenciesCsv.sh" or "externalDependenciesPython.sh" is required to run prior to this script. + +# Requires executeQueryFunctions.sh, cleanupAfterReportGeneration.sh + +# Fail on any error ("-e" = exit on first error, "-o pipefail" exist on errors within piped commands) +set -o errexit -o pipefail + +# Overrideable Constants (defaults also defined in sub scripts) +REPORTS_DIRECTORY=${REPORTS_DIRECTORY:-"reports"} +MARKDOWN_INCLUDES_DIRECTORY=${MARKDOWN_INCLUDES_DIRECTORY:-"includes"} # Subdirectory that contains Markdown files to be included by the Markdown template for the report. + +## Get this "domains/external-dependencies/summary" directory if not already set +# Even if $BASH_SOURCE is made for Bourne-like shells it is also supported by others and therefore here the preferred solution. +# CDPATH reduces the scope of the cd command to potentially prevent unintended directory changes. +# This way non-standard tools like readlink aren't needed. +EXTERNAL_DEPENDENCIES_SUMMARY_DIR=${EXTERNAL_DEPENDENCIES_SUMMARY_DIR:-$(CDPATH=. cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)} +#echo "externalDependenciesSummary: EXTERNAL_DEPENDENCIES_SUMMARY_DIR=${EXTERNAL_DEPENDENCIES_SUMMARY_DIR}" + +# Get the "scripts" directory by navigating three levels up from this summary directory. +SCRIPTS_DIR=${SCRIPTS_DIR:-"${EXTERNAL_DEPENDENCIES_SUMMARY_DIR}/../../../scripts"} +MARKDOWN_SCRIPTS_DIR=${MARKDOWN_SCRIPTS_DIR:-"${SCRIPTS_DIR}/markdown"} + +# Cypher query directory within this domain +EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR=${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR:-"${EXTERNAL_DEPENDENCIES_SUMMARY_DIR}/../queries"} + +# Define functions to execute a cypher query from within a given file (first and only argument) like "execute_cypher" +source "${SCRIPTS_DIR}/executeQueryFunctions.sh" + +# ── Front matter ────────────────────────────────────────────────────────────── + +external_dependencies_front_matter() { + local current_date + current_date="$(date +'%Y-%m-%d')" + + local latest_tag + latest_tag="$(git for-each-ref --sort=-creatordate --count=1 --format '%(refname:short)' refs/tags)" + + local analysis_directory + analysis_directory="${PWD##*/}" + + echo "---" + echo "title: \"External Dependencies Report\"" + echo "generated: \"${current_date}\"" + echo "model_version: \"${latest_tag}\"" + echo "dataset: \"${analysis_directory}\"" + echo "authors: [\"JohT/code-graph-analysis-pipeline\"]" + echo "---" +} + +# ── SVG chart reference helpers ─────────────────────────────────────────────── + +# Emits a Markdown image reference for a chart SVG if the file exists, otherwise nothing. +include_svg_if_exists() { + local svg_file="${FULL_REPORT_DIRECTORY}/${1}" + local alt_text="${2}" + if [ -f "${svg_file}" ]; then + echo "" + echo "![${alt_text}](./${1})" + echo "" + fi +} + +# Emits a Markdown image reference for every SVG matching the given glob pattern, sorted. +include_svgs_matching() { + local pattern="${1}" + find "${FULL_REPORT_DIRECTORY}" -maxdepth 1 -type f -name "${pattern}" | sort | while read -r svg_file; do + local chart_filename + chart_filename=$(basename -- "${svg_file}") + local chart_label="${chart_filename%.*}" + echo "" + echo "![${chart_label}](./${chart_filename})" + done +} + +# ── Report assembly ─────────────────────────────────────────────────────────── + +assemble_external_dependencies_report() { + echo "externalDependenciesSummary: $(date +'%Y-%m-%dT%H:%M:%S%z') Assembling Markdown report..." + + local report_include_directory="${FULL_REPORT_DIRECTORY}/${MARKDOWN_INCLUDES_DIRECTORY}" + mkdir -p "${report_include_directory}" + + # -- Write front matter ------------------------------------------------ + external_dependencies_front_matter > "${report_include_directory}/ExternalDependenciesReportFrontMatter.md" + + # -- Java: top external packages by caller types ----------------------- + execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_package_usage_overall.cypher" \ + --output-markdown-table \ + > "${report_include_directory}/External_package_usage_overall.md" + + # -- Java: second-level package grouping (top 20) --------------------- + execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_second_level_package_usage_overall.cypher" \ + --output-markdown-table \ + > "${report_include_directory}/External_second_level_package_usage_overall.md" + + # -- Java: most spread external packages ------------------------------ + execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_package_usage_spread.cypher" \ + --output-markdown-table \ + > "${report_include_directory}/External_package_usage_spread.md" + + # -- Java: external package usage per artifact (sorted) --------------- + execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_package_usage_per_artifact_sorted_top.cypher" \ + --output-markdown-table \ + > "${report_include_directory}/External_package_usage_per_artifact_sorted_top.md" + + # -- Java: aggregated external package usage per artifact ------------- + execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_package_usage_per_artifact_package_aggregated.cypher" \ + --output-markdown-table \ + > "${report_include_directory}/External_package_usage_per_artifact_package_aggregated.md" + + # -- TypeScript: top external modules --------------------------------- + execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_module_usage_overall_for_Typescript.cypher" \ + --output-markdown-table \ + > "${report_include_directory}/External_module_usage_overall_for_Typescript.md" + + # -- TypeScript: top external namespaces ------------------------------ + execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_namespace_usage_overall_for_Typescript.cypher" \ + --output-markdown-table \ + > "${report_include_directory}/External_namespace_usage_overall_for_Typescript.md" + + # -- TypeScript: module spread ---------------------------------------- + execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_module_usage_spread_for_Typescript.cypher" \ + --output-markdown-table \ + > "${report_include_directory}/External_module_usage_spread_for_Typescript.md" + + # -- TypeScript: external module usage per internal module (sorted) --- + execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/External_module_usage_per_internal_module_sorted_for_Typescript.cypher" \ + --output-markdown-table \ + > "${report_include_directory}/External_module_usage_per_internal_module_sorted_for_Typescript.md" + + # -- Maven POM declared dependencies ----------------------------------- + execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/Maven_POMs_and_their_declared_dependencies.cypher" \ + --output-markdown-table \ + > "${report_include_directory}/Maven_POM_dependencies.md" + + # -- Package.json dependency occurrence -------------------------------- + execute_cypher "${EXTERNAL_DEPENDENCIES_QUERY_CYPHER_DIR}/Package_json_dependencies_occurrence.cypher" \ + --output-markdown-table \ + > "${report_include_directory}/Package_json_dependencies_occurrence.md" + + # -- Inline SVG chart references (Java) -------------------------------- + { + include_svgs_matching "Java_Top_external_packages_by_types_above_threshold.svg" + include_svgs_matching "Java_Top_external_packages_by_types_others_drilldown.svg" + include_svgs_matching "Java_Top_external_packages_by_packages_above_threshold.svg" + include_svgs_matching "Java_Top_external_packages_by_packages_others_drilldown.svg" + include_svgs_matching "Java_Top_second_level_packages_by_types_above_threshold.svg" + include_svgs_matching "Java_Top_second_level_packages_by_packages_above_threshold.svg" + include_svgs_matching "Java_Most_spread_packages_by_types_above_threshold.svg" + include_svgs_matching "Java_Most_spread_packages_by_packages_above_threshold.svg" + include_svgs_matching "Java_Most_spread_second_level_packages_by_types_above_threshold.svg" + include_svgs_matching "Java_Most_spread_second_level_packages_by_packages_above_threshold.svg" + include_svgs_matching "Java_External_package_usage_per_artifact_stacked.svg" + include_svgs_matching "Java_External_second_level_package_usage_per_artifact_stacked.svg" + include_svgs_matching "Java_External_package_usage_max_internal_packages_percent.svg" + include_svgs_matching "Java_External_package_usage_median_internal_packages_percent.svg" + } > "${report_include_directory}/JavaExternalDependencyCharts.md" + + # -- Inline SVG chart references (TypeScript) --------------------------- + { + include_svgs_matching "Typescript_Top_external_modules_by_elements_above_threshold.svg" + include_svgs_matching "Typescript_Top_external_modules_by_elements_others_drilldown.svg" + include_svgs_matching "Typescript_Top_external_modules_by_modules_above_threshold.svg" + include_svgs_matching "Typescript_Top_external_namespaces_by_elements_above_threshold.svg" + include_svgs_matching "Typescript_Top_external_namespaces_by_modules_above_threshold.svg" + include_svgs_matching "Typescript_Most_spread_modules_by_declarations_above_threshold.svg" + include_svgs_matching "Typescript_Most_spread_modules_by_modules_above_threshold.svg" + include_svgs_matching "Typescript_Most_spread_namespaces_by_declarations_above_threshold.svg" + include_svgs_matching "Typescript_Most_spread_namespaces_by_modules_above_threshold.svg" + } > "${report_include_directory}/TypescriptExternalDependencyCharts.md" + + # -- Remove empty Markdown includes ------------------------------------ + source "${SCRIPTS_DIR}/cleanupAfterReportGeneration.sh" "${report_include_directory}" + + # -- Create fallback empty file for optional includes -------------------- + echo "" > "${report_include_directory}/empty.md" + + # -- Assemble final report from template -------------------------------- + cp -f "${EXTERNAL_DEPENDENCIES_SUMMARY_DIR}/report.template.md" "${FULL_REPORT_DIRECTORY}/report.template.md" + cat "${FULL_REPORT_DIRECTORY}/report.template.md" \ + | "${MARKDOWN_SCRIPTS_DIR}/embedMarkdownIncludes.sh" "${report_include_directory}" \ + > "${FULL_REPORT_DIRECTORY}/external_dependencies_report.md" + + rm -rf "${FULL_REPORT_DIRECTORY}/report.template.md" + rm -rf "${report_include_directory}" + + echo "externalDependenciesSummary: $(date +'%Y-%m-%dT%H:%M:%S%z') Successfully finished." +} + +# ── Main ────────────────────────────────────────────────────────────────────── + +# Create report directory +REPORT_NAME="external-dependencies" +FULL_REPORT_DIRECTORY="${REPORTS_DIRECTORY}/${REPORT_NAME}" +mkdir -p "${FULL_REPORT_DIRECTORY}" + +assemble_external_dependencies_report diff --git a/domains/external-dependencies/summary/report.template.md b/domains/external-dependencies/summary/report.template.md new file mode 100644 index 000000000..61206c9f0 --- /dev/null +++ b/domains/external-dependencies/summary/report.template.md @@ -0,0 +1,133 @@ + + +# πŸ“¦ External Dependencies Report + +## 1. Executive Overview + +This report analyses which **external packages, modules, and namespaces** the codebase depends on, how widely those dependencies are spread across internal modules, and how dependency usage breaks down per artifact. + +> **Who imports whom?** +> External dependencies reveal coupling to third-party libraries, infrastructure frameworks, and platform APIs. +> High spread (many internal modules referencing the same external package) indicates a candidate for a **facade** or **anti-corruption layer**. + +## πŸ“š Table of Contents + +1. [Executive Overview](#1-executive-overview) +1. [Java External Dependencies](#2-java-external-dependencies) +1. [TypeScript External Dependencies](#3-typescript-external-dependencies) +1. [Package Management](#4-package-management) +1. [Glossary](#5-glossary) + +--- + +## 2. Java External Dependencies + +### 2.1 Most Used External Packages + +The table below lists external Java packages ranked by how many *internal types* call into them. +High `numberOfExternalCallerTypes` values indicate packages that are deeply embedded in the codebase. + + + +### 2.2 Most Used External Packages β€” Second-Level Grouping + +Second-level package names (e.g. `javax.xml` from `javax.xml.stream`) reveal **framework-level** coupling that is hidden when viewing full package names. + + + +### 2.3 Most Spread External Packages + +Packages referenced from the highest number of **different internal packages**. +High spread indicates a pervasive cross-cutting dependency that is hard to replace. + + + +### 2.4 External Package Usage per Artifact (Top) + +Which artifacts are most exposed to external packages? +Ranked by total number of internal packages with external dependencies. + + + +### 2.5 Aggregated External Package Usage per Artifact + +Statistical summary: for each artifact, how many internal packages use external ones, and what percentage is that? + + + +### 2.6 Java Charts + + + +--- + +## 3. TypeScript External Dependencies + +### 3.1 Most Used External Modules + +External npm modules ranked by how many internal TypeScript elements import them. + + + +### 3.2 Most Used External Namespaces + +Some npm packages expose multiple namespaces. This view groups by namespace to show +declaration-level coupling within a package. + + + +### 3.3 Most Spread External Modules + +External modules referenced from the highest number of **different internal TypeScript modules**. + + + +### 3.4 External Module Usage per Internal Module (Sorted) + +Which internal TypeScript modules depend on the most external modules? + + + +### 3.5 TypeScript Charts + + + +--- + +## 4. Package Management + +### 4.1 Maven POM Declared Dependencies + +Java dependencies declared in Maven `pom.xml` files β€” declared vs. analysed. + + + +### 4.2 package.json Declared Dependencies + +TypeScript / Node.js dependencies declared in `package.json` files, ranked by occurrence +across repository packages. + + + +--- + +## 5. Glossary + +| Term | Definition | +|------|-----------| +| **External dependency** | A type, module, or namespace whose source code is *not* part of the scanned internal codebase. | +| **External package** | A fully-qualified Java package belonging to an external library (e.g. `org.springframework.web`). | +| **Second-level package** | The first two dot-separated segments of a package name (e.g. `org.springframework`). Used to group many variant packages under one framework name. | +| **External module** | A TypeScript/JavaScript npm package imported via `import ... from 'module-name'`. | +| **External namespace** | A named export group within an external TypeScript module (e.g. `node:fs`, `@angular/core`). | +| **numberOfExternalCallerTypes** | Count of *internal Java types* that directly reference the external package. | +| **numberOfExternalCallerPackages** | Count of *internal Java packages* that contain at least one type referencing the external package. | +| **numberOfExternalCallerElements** | Count of *internal TypeScript elements* (functions, classes, variables) that import from the external module. | +| **numberOfExternalCallerModules** | Count of *internal TypeScript modules* that import from the external module. | +| **sumNumberOfTypes / sumNumberOfPackages** | Sum across all artifacts of internal types/packages dependent on the external package. Measures total spread. | +| **sumNumberOfUsedExternalDeclarations** | Total distinct external declarations (functions, types) imported from the module across all internal modules. | +| **packagesCallingExternalRate** | Ratio of internal packages with external references to total artifact packages. | +| **typesCallingExternalRate** | Ratio of internal types with external references to total artifact types. | +| **Anti-Corruption Layer (ACL)** | An architectural pattern that isolates the internal domain from external library APIs by wrapping them behind an internal interface. Recommended when a high-spread dependency must be replaced or abstracted. | +| **FaΓ§ade** | A simplified interface that hides the complexity of one or more external libraries. Useful when many internal modules call the same external package. | +| **Hexagonal Architecture** | An architectural style (Ports & Adapters) that pushes all external dependencies to the outer ring, preventing them from leaking into the core domain. | diff --git a/scripts/executeQuery.sh b/scripts/executeQuery.sh index 04b53766c..4b252c0e9 100755 --- a/scripts/executeQuery.sh +++ b/scripts/executeQuery.sh @@ -159,7 +159,8 @@ else if [ "${no_source_reference}" = true ] ; then echo -n "${cypher_query_result}" | jq -r '(.results[0])? | .columns,(.data[].row)? | map(if type == "array" then join(",") else . end) | flatten | @csv' else - cypher_query_file_relative_name=${cypher_query_file_name#/**/cypher/} + repository_root="$(CDPATH=. cd -- "${SCRIPTS_DIR}/.." && pwd -P)" + cypher_query_file_relative_name="${cypher_query_file_name#${repository_root}/}" sourceFileReferenceInfo="Source Cypher File: ${cypher_query_file_relative_name}" echo -n "${cypher_query_result}" | jq -r --arg sourceReference "${sourceFileReferenceInfo}" '(.results[0])? | .columns + [$sourceReference], (.data[].row)? + [""] | map(if type == "array" then join(",") else . end) | flatten | @csv' fi diff --git a/scripts/prepareAnalysis.sh b/scripts/prepareAnalysis.sh index e623b26c7..62485b9b9 100644 --- a/scripts/prepareAnalysis.sh +++ b/scripts/prepareAnalysis.sh @@ -36,7 +36,7 @@ source "${SCRIPTS_DIR}/parseCsvFunctions.sh" # Local Constants DEPENDS_ON_CYPHER_DIR="$CYPHER_DIR/DependsOn_Relationship_Weights" METRICS_CYPHER_DIR="$CYPHER_DIR/Metrics" -EXTERNAL_DEPENDENCIES_CYPHER_DIR="$CYPHER_DIR/External_Dependencies" +JAVA_CYPHER_DIR="$CYPHER_DIR/Java" ARTIFACT_DEPENDENCIES_CYPHER_DIR="$CYPHER_DIR/Artifact_Dependencies" TYPES_CYPHER_DIR="$CYPHER_DIR/Types" TYPESCRIPT_CYPHER_DIR="$CYPHER_DIR/Typescript_Enrichment" @@ -138,8 +138,8 @@ execute_cypher "${TYPES_CYPHER_DIR}/Label_base_java_types.cypher" execute_cypher "${TYPES_CYPHER_DIR}/Label_buildin_java_types.cypher" execute_cypher "${TYPES_CYPHER_DIR}/Label_resolved_duplicate_types.cypher" -execute_cypher "${EXTERNAL_DEPENDENCIES_CYPHER_DIR}/Remove_external_type_and_annotation_labels.cypher" -execute_cypher "${EXTERNAL_DEPENDENCIES_CYPHER_DIR}/Label_external_types_and_annotations.cypher" +execute_cypher "${JAVA_CYPHER_DIR}/Remove_external_type_and_annotation_labels.cypher" +execute_cypher "${JAVA_CYPHER_DIR}/Label_external_types_and_annotations.cypher" # Preparation - Add Java Artifact node properties "incomingDependencies", "outgoingDependencies" and "version" execute_cypher_summarized "${ARTIFACT_DEPENDENCIES_CYPHER_DIR}/Incoming_Java_Artifact_Dependencies.cypher" diff --git a/scripts/reports/ExternalDependenciesCsv.sh b/scripts/reports/ExternalDependenciesCsv.sh deleted file mode 100755 index 5f61f42e6..000000000 --- a/scripts/reports/ExternalDependenciesCsv.sh +++ /dev/null @@ -1,81 +0,0 @@ -#!/usr/bin/env bash - -# Executes "External_Dependencies" Cypher queries to get the "external-dependencies-csv" CSV reports. -# They list external library package usage like how often a external package is called. - -# Requires executeQueryFunctions.sh, cleanupAfterReportGeneration.sh - -# Fail on any error ("-e" = exit on first error, "-o pipefail" exist on errors within piped commands) -set -o errexit -o pipefail - -# Overrideable Constants (defaults also defined in sub scripts) -REPORTS_DIRECTORY=${REPORTS_DIRECTORY:-"reports"} - -## Get this "scripts/reports" directory if not already set -# Even if $BASH_SOURCE is made for Bourne-like shells it is also supported by others and therefore here the preferred solution. -# CDPATH reduces the scope of the cd command to potentially prevent unintended directory changes. -# This way non-standard tools like readlink aren't needed. -REPORTS_SCRIPT_DIR=${REPORTS_SCRIPT_DIR:-$( CDPATH=. cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P )} -echo "ExternalDependenciesCsv: REPORTS_SCRIPT_DIR=${REPORTS_SCRIPT_DIR}" - -# Get the "scripts" directory by taking the path of this script and going one directory up. -SCRIPTS_DIR=${SCRIPTS_DIR:-"${REPORTS_SCRIPT_DIR}/.."} # Repository directory containing the shell scripts -echo "ExternalDependenciesCsv SCRIPTS_DIR=${SCRIPTS_DIR}" - -# Get the "cypher" directory by taking the path of this script and going two directory up and then to "cypher". -CYPHER_DIR=${CYPHER_DIR:-"${REPORTS_SCRIPT_DIR}/../../cypher"} -echo "ExternalDependenciesCsv CYPHER_DIR=${CYPHER_DIR}" - -# Define functions to execute cypher queries from within a given file -source "${SCRIPTS_DIR}/executeQueryFunctions.sh" - -# Create report directory -REPORT_NAME="external-dependencies-csv" -FULL_REPORT_DIRECTORY="${REPORTS_DIRECTORY}/${REPORT_NAME}" -mkdir -p "${FULL_REPORT_DIRECTORY}" - -# Local Constants -EXTERNAL_DEPENDENCIES_CYPHER_DIR="${CYPHER_DIR}/External_Dependencies" - -# Check if there are already labels for external Java types and create them otherwise -execute_cypher_queries_until_results "${EXTERNAL_DEPENDENCIES_CYPHER_DIR}/List_external_Java_types_used.cypher" \ - "${EXTERNAL_DEPENDENCIES_CYPHER_DIR}/Label_external_types_and_annotations.cypher" >/dev/null - -# CSV reports for Java Packages - -execute_cypher "${EXTERNAL_DEPENDENCIES_CYPHER_DIR}/External_package_usage_overall.cypher" > "${FULL_REPORT_DIRECTORY}/External_package_usage_overall.csv" -execute_cypher "${EXTERNAL_DEPENDENCIES_CYPHER_DIR}/External_package_usage_spread.cypher" > "${FULL_REPORT_DIRECTORY}/External_package_usage_spread.csv" -execute_cypher "${EXTERNAL_DEPENDENCIES_CYPHER_DIR}/External_second_level_package_usage_overall.cypher" > "${FULL_REPORT_DIRECTORY}/External_second_level_package_usage_overall.csv" -execute_cypher "${EXTERNAL_DEPENDENCIES_CYPHER_DIR}/External_second_level_package_usage_spread.cypher" > "${FULL_REPORT_DIRECTORY}/External_second_level_package_usage_spread.csv" -execute_cypher "${EXTERNAL_DEPENDENCIES_CYPHER_DIR}/External_package_usage_per_type.cypher" > "${FULL_REPORT_DIRECTORY}/External_package_usage_per_type.csv" -execute_cypher "${EXTERNAL_DEPENDENCIES_CYPHER_DIR}/External_package_usage_per_artifact.cypher" > "${FULL_REPORT_DIRECTORY}/External_package_usage_per_artifact.csv" -execute_cypher "${EXTERNAL_DEPENDENCIES_CYPHER_DIR}/External_package_usage_per_artifact_sorted_top.cypher" > "${FULL_REPORT_DIRECTORY}/External_package_usage_per_artifact_sorted_top.csv" -execute_cypher "${EXTERNAL_DEPENDENCIES_CYPHER_DIR}/External_package_usage_per_artifact_distribution.cypher" > "${FULL_REPORT_DIRECTORY}/External_package_usage_per_artifact_distribution.csv" -execute_cypher "${EXTERNAL_DEPENDENCIES_CYPHER_DIR}/External_package_usage_per_artifact_package_aggregated.cypher" > "${FULL_REPORT_DIRECTORY}/External_package_usage_per_artifact_package_aggregated.csv" -execute_cypher "${EXTERNAL_DEPENDENCIES_CYPHER_DIR}/External_package_usage_per_artifact_and_package.cypher" > "${FULL_REPORT_DIRECTORY}/External_package_usage_per_artifact_and_package.csv" -execute_cypher "${EXTERNAL_DEPENDENCIES_CYPHER_DIR}/External_package_usage_per_artifact_and_external_package.cypher" > "${FULL_REPORT_DIRECTORY}/External_package_usage_per_artifact_and_external_package.csv" -execute_cypher "${EXTERNAL_DEPENDENCIES_CYPHER_DIR}/External_second_level_package_usage_per_artifact_and_external_package.cypher" > "${FULL_REPORT_DIRECTORY}/External_second_level_package_usage_per_artifact_and_external_package.csv" - -execute_cypher "${EXTERNAL_DEPENDENCIES_CYPHER_DIR}/Maven_POMs_and_their_declared_dependencies.cypher" > "${FULL_REPORT_DIRECTORY}/Maven_POM_dependencies.csv" - -# CSV reports for Typescript Modules - -execute_cypher "${EXTERNAL_DEPENDENCIES_CYPHER_DIR}/External_module_usage_overall_for_Typescript.cypher" > "${FULL_REPORT_DIRECTORY}/External_module_usage_overall_for_Typescript.csv" -execute_cypher "${EXTERNAL_DEPENDENCIES_CYPHER_DIR}/External_module_usage_spread_for_Typescript.cypher" > "${FULL_REPORT_DIRECTORY}/External_module_usage_spread_for_Typescript.csv" - -execute_cypher "${EXTERNAL_DEPENDENCIES_CYPHER_DIR}/External_namespace_usage_overall_for_Typescript.cypher" > "${FULL_REPORT_DIRECTORY}/External_namespace_usage_overall_for_Typescript.csv" -execute_cypher "${EXTERNAL_DEPENDENCIES_CYPHER_DIR}/External_namespace_usage_spread_for_Typescript.cypher" > "${FULL_REPORT_DIRECTORY}/External_namespace_usage_spread_for_Typescript.csv" - -execute_cypher "${EXTERNAL_DEPENDENCIES_CYPHER_DIR}/External_module_usage_per_internal_module_sorted_for_Typescript.cypher" > "${FULL_REPORT_DIRECTORY}/External_module_usage_per_internal_module_sorted_for_Typescript.csv" -execute_cypher "${EXTERNAL_DEPENDENCIES_CYPHER_DIR}/External_namespace_usage_per_internal_module_sorted_for_Typescript.cypher" > "${FULL_REPORT_DIRECTORY}/External_namespace_usage_per_internal_module_sorted_for_Typescript.csv" - -execute_cypher "${EXTERNAL_DEPENDENCIES_CYPHER_DIR}/External_module_usage_per_internal_module_aggregated_for_Typescript.cypher" > "${FULL_REPORT_DIRECTORY}/External_module_usage_per_internal_module_aggregated_for_Typescript.csv" -execute_cypher "${EXTERNAL_DEPENDENCIES_CYPHER_DIR}/External_module_usage_per_internal_module_distribution_for_Typescript.cypher" > "${FULL_REPORT_DIRECTORY}/External_module_usage_per_internal_module_distribution_for_Typescript.csv" - -execute_cypher "${EXTERNAL_DEPENDENCIES_CYPHER_DIR}/List_external_modules_resolved_to_internal_ones_for_Typescript.cypher" > "${FULL_REPORT_DIRECTORY}/External_modules_resolved_to_internal_ones_for_Typescript.csv" - -execute_cypher "${EXTERNAL_DEPENDENCIES_CYPHER_DIR}/Package_json_dependencies_occurrence.cypher" > "${FULL_REPORT_DIRECTORY}/Package_json_dependencies_occurrence.csv" -execute_cypher "${EXTERNAL_DEPENDENCIES_CYPHER_DIR}/Package_json_dependencies_combinations.cypher" > "${FULL_REPORT_DIRECTORY}/Package_json_dependencies_combinations.csv" - -# Clean-up after report generation. Empty reports will be deleted. -source "${SCRIPTS_DIR}/cleanupAfterReportGeneration.sh" "${FULL_REPORT_DIRECTORY}" \ No newline at end of file