diff --git a/docs/source/basics/introduction.rst b/docs/source/basics/introduction.rst index 1004513..e29aa71 100644 --- a/docs/source/basics/introduction.rst +++ b/docs/source/basics/introduction.rst @@ -1,14 +1,37 @@ Introduction ============ -``CodeLinks`` is a Sphinx extension that provides the ``src-trace`` directive to establish traceability between source code and :external+needs:doc:`Sphinx-Needs ` items. +``CodeLinks`` is a versatile utility that enables fast source code tracing and connects it to +the :external+needs:doc:`Sphinx-Needs ` ecosystem. -At its core, ``CodeLinks`` uses a powerful ``Analyse`` to parse source code comments and extract valuable information. The ``Analyse`` can identify and extract three distinct types of content: +It has multiple components: -- **One-line need definitions**: Create new Sphinx-Needs directly from a single, :ref:`customized comment line ` in your source code. +- source code analyzer for multiple programming languages and comment styles +- generator for various output formats that contain the extracted markers or needs +- Sphinx extension that integrates ``CodeLinks`` with Sphinx-Needs +- CLI application to drive the analysis and generation process + +The configuration for ``CodeLinks`` is done via a TOML file, which can be used +for both the :ref:`Sphinx extension ` and the :ref:`CLI application `. + +The configuration determines how markers and languages are ingested and how the Sphinx extension should behave. + +At its core, ``CodeLinks`` parses the source code structure and extracts source markers. +Source markers can be special comments or language-specific constructs like docstrings for Python. + +``CodeLinks`` supports 3 distinct marker types: + +- :ref:`One-line need definitions `: Create new Sphinx-Needs directly from a single customized comment line + in your source code. - **Need ID references**: Link code to existing need items without creating new ones, perfect for tracing implementations to requirements. - **Marked RST text**: Extract blocks of reStructuredText embedded within comments, allowing you to include rich documentation with associated metadata right next to your code. -``src-trace`` directive then consumes ``One-line need definitions`` to generate traceability between source code and your documentation. +When used in a Sphinx context, a new :ref:`directive` creates items at the location where it is placed (for a subset +of the analyzed files/folders). + +For use cases where ``CodeLinks`` should not integrate with Sphinx, but rather generate output files, the +:ref:`cli` can be used. Currently it can write out ``needextend`` directives for the need ID reference comment style. +Other output files are planned such as full need items as RST or needs.json. -The ``Analyse``, along with the ``SourceDiscovery`` module, also provides both a **Python API** for extensibility and a **CLI** for integration into CI/CD pipelines. +The availability of most commands as :ref:`cli` also helps integrate ``CodeLinks`` into build systems and CI/CD pipelines. +Focus is put on performance, portability and caching of processing steps. diff --git a/docs/source/basics/quickstart.rst b/docs/source/basics/quickstart.rst index 54f0740..7dec73c 100644 --- a/docs/source/basics/quickstart.rst +++ b/docs/source/basics/quickstart.rst @@ -26,7 +26,7 @@ Sphinx Config .. literalinclude:: ./../../src_trace.toml :caption: src_trace.toml :language: toml - :lines: 27-29 + One-line comment ---------------- diff --git a/docs/source/components/analyse.rst b/docs/source/components/analyse.rst index 8cd9e04..4841451 100644 --- a/docs/source/components/analyse.rst +++ b/docs/source/components/analyse.rst @@ -1,27 +1,62 @@ -Analyse -======= +.. _analyse: -The ``Analyse`` is a :ref:`CLI tool ` that also provides an API for programmatic use. Its primary function is to extract specific, marked content from comments within source code files. +Source Analyse +============== -It can extract three types of content: +The **Source Analyse** module is a powerful component of **Sphinx-CodeLinks** that extracts documentation-related content from source code comments. It provides both CLI and API interfaces for flexible integration into documentation workflows. -- Sphinx-Needs ID References -- Oneline needs (see :ref:`OneLineCommentStyle `) -- Marked reStructuredText (RST) blocks +**Key Capabilities:** -Configuration -------------- +- Extract **Sphinx-Needs** ID references from source code comments +- Process custom one-line comment patterns for rapid documentation +- Extract marked reStructuredText (RST) blocks embedded in comments +- Generate structured JSON output for further processing +- Support for multiple programming language comment styles -The ``Analyse`` is configured using a ``toml`` file. The examples throughout this document are based on the following configuration: +Overview +-------- -.. literalinclude:: ./../../../tests/data/analyse/default_config.toml - :caption: default_config.toml - :language: toml +Source Analyse works by parsing source code files and identifying specially marked comments that contain documentation information. This enables developers to embed documentation directly in their source code while maintaining clean separation between code and documentation. -This configuration instructs the analyse to extract ``Sphinx-Needs ID Refs`` and ``Marked rst text`` using the defined markers. +The module supports three primary extraction modes: -Sphinx-Needs ID Refs --------------------- +1. **Sphinx-Needs ID References** - Links between code and requirements/specifications +2. **One-line Needs** - Simplified syntax for creating documentation needs +3. **Marked RST Blocks** - Full reStructuredText content embedded in comments + +Supported Content Types +----------------------- + +Sphinx-Needs ID References +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Extract references to **Sphinx-Needs** items directly from source code comments, enabling traceability between code implementations and requirements. + +One-line Needs +~~~~~~~~~~~~~~ + +Use simplified comment patterns to define **Sphinx-Needs** items without complex RST syntax. See :ref:`OneLineCommentStyle ` for detailed information. + +Marked RST Blocks +~~~~~~~~~~~~~~~~~ + +Embed complete reStructuredText content within source code comments for rich documentation that can be extracted and processed. + +Limitations +----------- + +**Current Limitations:** + +- **Language Support**: Only C/C++ (``//``, ``/* */``) and Python (``#``) comment styles are supported +- **Single Comment Style**: Each analysis run processes only one comment style at a time + +Extraction Examples +------------------- + +The following examples are configured with :ref:`the analyse configuration `, + +Sphinx-Needs ID References +~~~~~~~~~~~~~~~~~~~~~~~~~~ Below is an example of a C++ source file containing need ID references and the corresponding JSON output from the analyse. @@ -29,52 +64,62 @@ Below is an example of a C++ source file containing need ID references and the c .. code-tab:: cpp - #include + #include - // @need-ids: need_001, need_002, need_003, need_004 - void dummy_func1(){ - //... - } + // @need-ids: need_001, need_002, need_003, need_004 + void dummy_func1(){ + //... + } - // @need-ids: need_003 - int main() { - std::cout << "Starting demo_1..." << std::endl; - dummy_func1(); - std::cout << "Demo_1 finished." << std::endl; - return 0; - } + // @need-ids: need_003 + int main() { + std::cout << "Starting demo_1..." << std::endl; + dummy_func1(); + std::cout << "Demo_1 finished." << std::endl; + return 0; + } .. code-tab:: json - [ - { - "filepath": "tests/data/need_id_refs/dummy_1.cpp", - "remote_url": "https://github.com/useblocks/sphinx-codelinks/blob/fa5a9129d60203355ae9fe4a725246a88522c60c/tests/data/need_id_refs/dummy_1.cpp#L3", - "source_map": { - "start": { "row": 2, "column": 13 }, - "end": { "row": 2, "column": 51 } - }, - "tagged_scope": "void dummy_func1(){\n //...\n }", - "need_ids": ["need_001", "need_002", "need_003", "need_004"], - "marker": "@need-ids:", - "type": "need-id-refs" - }, - { - "filepath": "tests/data/need_id_refs/dummy_1.cpp", - "remote_url": "https://github.com/useblocks/sphinx-codelinks/blob/fa5a9129d60203355ae9fe4a725246a88522c60c/tests/data/need_id_refs/dummy_1.cpp#L8", - "source_map": { - "start": { "row": 7, "column": 13 }, - "end": { "row": 7, "column": 21 } - }, - "tagged_scope": "int main() {\n std::cout << \"Starting demo_1...\" << std::endl;\n dummy_func1();\n std::cout << \"Demo_1 finished.\" << std::endl;\n return 0;\n }", - "need_ids": ["need_003"], - "marker": "@need-ids:", - "type": "need-id-refs" - } - ] - -Marked RST ----------- + [ + { + "filepath": "tests/data/need_id_refs/dummy_1.cpp", + "remote_url": "https://github.com/useblocks/sphinx-codelinks/blob/fa5a9129d60203355ae9fe4a725246a88522c60c/tests/data/need_id_refs/dummy_1.cpp#L3", + "source_map": { + "start": { "row": 2, "column": 13 }, + "end": { "row": 2, "column": 51 } + }, + "tagged_scope": "void dummy_func1(){\n //...\n }", + "need_ids": ["need_001", "need_002", "need_003", "need_004"], + "marker": "@need-ids:", + "type": "need-id-refs" + }, + { + "filepath": "tests/data/need_id_refs/dummy_1.cpp", + "remote_url": "https://github.com/useblocks/sphinx-codelinks/blob/fa5a9129d60203355ae9fe4a725246a88522c60c/tests/data/need_id_refs/dummy_1.cpp#L8", + "source_map": { + "start": { "row": 7, "column": 13 }, + "end": { "row": 7, "column": 21 } + }, + "tagged_scope": "int main() {\n std::cout << \"Starting demo_1...\" << std::endl;\n dummy_func1();\n std::cout << \"Demo_1 finished.\" << std::endl;\n return 0;\n }", + "need_ids": ["need_003"], + "marker": "@need-ids:", + "type": "need-id-refs" + } + ] + +**Output Structure:** + +- ``filepath`` - Path to the source file containing the reference +- ``remote_url`` - URL to the source code in the remote repository +- ``source_map`` - Location information (row/column) of the marker +- ``tagged_scope`` - The code scope associated with the marker +- ``need_ids`` - List of referenced need IDs +- ``marker`` - The marker string used for identification +- ``type`` - Type of extraction ("need-id-refs") + +Marked RST Blocks +~~~~~~~~~~~~~~~~~ This example demonstrates how the analyse extracts RST blocks from comments. @@ -102,7 +147,6 @@ This example demonstrates how the analyse extracts RST blocks from comments. return 0; } - .. code-tab:: json [ @@ -110,8 +154,8 @@ This example demonstrates how the analyse extracts RST blocks from comments. "filepath": "marked_rst/dummy_1.cpp", "remote_url": "https://github.com/useblocks/sphinx-codelinks/blob/26b301138eef25c5130518d96eaa7a29a9c6c9fe/marked_rst/dummy_1.cpp#L4", "source_map": { - "start": { "row": 3, "column": 8 }, - "end": { "row": 3, "column": 61 } + "start": { "row": 3, "column": 8 }, + "end": { "row": 3, "column": 61 } }, "tagged_scope": "void dummy_func1(){\n //...\n }", "rst": ".. impl:: implement dummy function 1\n :id: IMPL_71\n", @@ -121,8 +165,8 @@ This example demonstrates how the analyse extracts RST blocks from comments. "filepath": "marked_rst/dummy_1.cpp", "remote_url": "https://github.com/useblocks/sphinx-codelinks/blob/26b301138eef25c5130518d96eaa7a29a9c6c9fe/marked_rst/dummy_1.cpp#L14", "source_map": { - "start": { "row": 13, "column": 7 }, - "end": { "row": 13, "column": 40 } + "start": { "row": 13, "column": 7 }, + "end": { "row": 13, "column": 40 } }, "tagged_scope": "int main() {\n std::cout << \"Starting demo_1...\" << std::endl;\n dummy_func1();\n std::cout << \"Demo_1 finished.\" << std::endl;\n return 0;\n }", "rst": "..impl:: implement main function ", @@ -130,11 +174,39 @@ This example demonstrates how the analyse extracts RST blocks from comments. } ] -Limitations ------------ +**Output Structure:** + +- ``filepath`` - Path to the source file containing the RST block +- ``remote_url`` - URL to the source code in the remote repository +- ``source_map`` - Location information of the RST markers +- ``tagged_scope`` - The code scope associated with the RST block +- ``rst`` - The extracted reStructuredText content +- ``type`` - Type of extraction ("rst") + +**RST Block Formats:** + +The module supports both multi-line and single-line RST blocks: + +- **Multi-line blocks**: Use ``@rst`` and ``@endrst`` on separate lines +- **Single-line blocks**: Use ``@rst content @endrst`` on the same line + +One-line Needs +-------------- + +**One-line Needs** provide a simplified syntax for creating **Sphinx-Needs** items directly in source code comments without requiring full RST syntax. + +For comprehensive information about one-line needs configuration and usage, see :ref:`OneLineCommentStyle `. + +**Basic Example:** + +.. code-block:: c + + // @Function Implementation, IMPL_001, impl, [REQ_001, REQ_002] + +This single comment line creates a complete **Sphinx-Needs** item equivalent to: -Please be aware of the following limitations: +.. code-block:: rst -- **Supported Languages**: The analyse only supports comment styles for C/C++ (``//``, ``/*...*/``) and Python (``#``). -- **Single Comment Style**: An analysis run can only process a single comment style at a time. -- **Configuration Incompatibility**: The TOML configuration file cannot be shared with the ``CodeLink`` Sphinx extensions. + .. impl:: Function Implementation + :id: IMPL_001 + :links: REQ_001, REQ_002 diff --git a/docs/source/components/cli.rst b/docs/source/components/cli.rst index dfa47bd..6d1d0bf 100644 --- a/docs/source/components/cli.rst +++ b/docs/source/components/cli.rst @@ -3,10 +3,10 @@ Command Line Interface (CLI) ============================ -``Sphinx-CodeLinks`` provides CLI for users to integrate documentation build into CI/CD pipeline +``Sphinx-CodeLinks`` provides a CLI for users to integrate documentation builds into CI/CD pipelines and for local development. -It features help pages. add ``-h`` or ``--help`` to any command to see the available options. +It features help pages. Add ``-h`` or ``--help`` to any command to see the available options. .. typer:: sphinx_codelinks.cmd.app :prog: codelinks diff --git a/docs/source/components/configuration.rst b/docs/source/components/configuration.rst index 751bb40..b078bf1 100644 --- a/docs/source/components/configuration.rst +++ b/docs/source/components/configuration.rst @@ -1,363 +1,488 @@ .. _configuration: -Configuration[Sphinx] -===================== +Configuration +============= The configuration for ``CodeLinks`` takes place in the project's :external+sphinx:ref:`conf.py file `. Each source code project may have different configurations because of its programming language or its locations. -Therefore, based on such consideration, there are **global options** and **project-specific options** for ``CodeLinks`` +Therefore, based on such considerations, there are **global options** and **project-specific options** for ``CodeLinks``. -All configuration options start with the prefix ``src_trace_`` for **Sphinx-CodeLinks**. +.. attention:: It is highly recommended to set the configuration options in a TOML file, which can be used for both the Sphinx extension and the CLI application. -Global Options --------------- +If the configurations are set in ``conf.py``, the options start with the prefix ``src_trace_``. -The options starts with the prefix ``src_trace_`` are globally applied in the scope of Sphinx documentation. +Sphinx Configuration +-------------------- + +In ``conf.py``, a TOML file can be specified as the source of the configuration for Sphinx Directive ``src-trace``. src_trace_config_from_toml ~~~~~~~~~~~~~~~~~~~~~~~~~~ -This configuration takes the (relative) path to a `toml file `__ -which contains some or all of the ``CodeLinks`` configuration -(configuration in the toml will override that in the :file:`conf.py`). +Specifies the path to a `TOML file `__ containing **Sphinx-CodeLinks** configuration options. This allows you to maintain configuration in a separate file for better organization. + +**Type:** ``str`` (relative path to the directory where conf.py is located) +**Default:** Not set .. code-block:: python - # Specify the config path for source tracing in conf.py + # In conf.py src_trace_config_from_toml = "src_trace.toml" -Configuration in the toml can contain any of the following options, under a ``[src_trace]`` section, -but with the ``src_trace_`` prefix removed. +When using a TOML configuration file: -.. caution:: Any configuration specifying relative paths in the toml file will be resolved relatively to the directory containing the ``toml`` file. +- Configuration options are placed under a ``[codelinks]`` section +- The ``src_trace_`` prefix is omitted in the TOML file +- TOML configuration overrides settings in :file:`conf.py` -.. _`src_trace_set_local_url`: +.. caution:: Relative paths specified in the TOML file are resolved relative to the directory containing the TOML file, not the Sphinx project root. -src_trace_set_local_url -~~~~~~~~~~~~~~~~~~~~~~~ +.. _`set_local_url`: -Set this option to ``True``, if the local link between a need to the local source code where it is defined is required. +Global Options +-------------- -Default: **False** +set_local_url +~~~~~~~~~~~~~ -.. tabs:: +Enables the generation of local file system links to source code locations. When enabled, Sphinx Directive **src-trace** will add a custom field, which contains the local path to the source file, to generated needs. - .. code-tab:: python +**Type:** ``bool`` +**Default:** ``False`` - src_trace_set_local_url = True +.. code-block:: toml - .. code-tab:: toml + [codelinks] + set_local_url = true - [src_trace] - set_local_url = true +local_url_field +~~~~~~~~~~~~~~~ -src_trace_set_local_field -~~~~~~~~~~~~~~~~~~~~~~~~~ +Specifies the custom field name used for local source code links. -.. note:: This option is only required if :ref:`src_trace_set_local_url` is set to **True**. +**Type:** ``str`` +**Default:** ``"local-url"`` +**Required when:** :ref:`set_local_url` is ``True`` -Set the desired custom field name for the local link to the source code. +.. code-block:: toml -Default: **local-url** + [codelinks] + local_url_field = "local-url" -.. tabs:: +.. _`set_remote_url`: - .. code-tab:: python +set_remote_url +~~~~~~~~~~~~~~ - src_trace_local_url_field = "local-url" +Enables the generation of remote repository links to source code locations. When enabled, Sphinx Directive **src-trace** will add a custom field, which contains the URL to the remote repository (e.g., GitHub, GitLab) where the source file is hosted, to needs. - .. code-tab:: toml +**Type:** ``bool`` +**Default:** ``False`` - [src_trace] - local_url_field = "local-url" +.. code-block:: toml -.. _`src_trace_set_remote_url`: + [codelinks] + set_remote_url = true -src_trace_set_remote_url -~~~~~~~~~~~~~~~~~~~~~~~~ +remote_url_field +~~~~~~~~~~~~~~~~ -Set this option to ``True``, if the remote link between a need to the remote source code -where it is defined is required. +Specifies the custom field name used for remote source code links. -The remote means where the source code is hosted such as GitHub. +**Type:** ``str`` +**Default:** ``"remote-url"`` +**Required when:** :ref:`set_remote_url` is ``True`` -Default: **False** +.. code-block:: toml -.. tabs:: + [codelinks] + remote_url_field = "remote-url" - .. code-tab:: python +outdir +~~~~~~ - src_trace_set_remote_url = True +Specifies the output directory for generated artifacts such as extracted markers and warnings. - .. code-tab:: toml +**Type:** ``str`` +**Default:** ``"./output"`` - [src_trace] - set_remote_url = true +.. code-block:: toml -src_trace_set_remote_field -~~~~~~~~~~~~~~~~~~~~~~~~~~ + [codelinks] + outdir = "output" -.. note:: This option is only required if :ref:`src_trace_set_remote_url` is set to **True**. +Project-Specific Options +------------------------ -Set the desired custom field name for the remote link to the source code. +Project-specific options are configured within the ``projects`` section, allowing different settings for :ref:`SourceDiscover ` and :ref:`SourceAnalyse `. -Default: **remote-url** +projects +~~~~~~~~ -.. tabs:: +Defines configuration for individual source code projects. Each project is identified by a unique name (key) and contains its own set of configuration options (value). - .. code-tab:: python +**Type:** ``dict[str, dict]`` +**Default:** ``{}`` - src_trace_remote_url_field = "remote-url" +.. code-block:: toml - .. code-tab:: toml + [codelinks.projects.my_project] + # Configuration for "my_project" - [src_trace] - remote_url_field = "remote-url" + [codelinks.projects.another_project] + # Configuration for "another_project" -Project Specific Options ------------------------- +remote_url_pattern +~~~~~~~~~~~~~~~~~~ -Options defined in **src_trace_projects** are project-specific. +Defines the URL pattern for Sphinx Directive ``src-trace`` to generate links to remote source code repositories (e.g., GitHub, GitLab). This pattern uses placeholders that are dynamically replaced with actual values. -src_trace_projects -~~~~~~~~~~~~~~~~~~ +**Type:** ``str`` +**Default:** Not set +**Required when:** :ref:`set_remote_url` is ``True`` -This option contains multiple sets of project-specific options. The project name is defined as the key in a dictionary -and its corresponding value is a dictionary containing the options specific to that project. +**Available placeholders:** -.. tabs:: +- ``{commit}`` - Git commit hash +- ``{path}`` - Relative path to the source file +- ``{line}`` - Line number in the source file - .. code-tab:: python +.. code-block:: toml - project_options = dict() - src_trace_projects = { - "project_name": project_options - } + [codelinks.projects.my_project] + remote_url_pattern = "https://github.com/user/repo/blob/{commit}/{path}#L{line}" - .. code-tab:: toml +**Common patterns:** - [src_trace.projects.project_name] - # Project configuration for "project_name" shall be written here +- **GitHub:** ``https://github.com/user/repo/blob/{commit}/{path}#L{line}`` +- **GitLab:** ``https://gitlab.com/user/repo/-/blob/{commit}/{path}#L{line}`` +- **Bitbucket:** ``https://bitbucket.org/user/repo/src/{commit}/{path}#lines-{line}`` -comment_type -~~~~~~~~~~~~ +.. note:: This option integrates with :external+needs:ref:`need_string_links` to automatically generate clickable links in the documentation. -This option defines the comment type used in the source code of the project. +.. _`discover_config`: -Default: **cpp** +source_discover +~~~~~~~~~~~~~~~ -.. note:: Currently, only C/C++ is supported +Configures how **Sphinx-CodeLinks** discovers and processes source files within a project. This option controls which files are analyzed for extracting documentation needs. -.. tabs:: +**Type:** ``dict`` +**Default:** See below - .. code-tab:: python +.. code-block:: toml - src_trace_projects = { - "project_name": { - "comment_type": "c" - } - } + [codelinks.projects.my_project.source_discover] + src_dir = "./" + exclude = [] + include = [] + gitignore = true + comment_type = "cpp" - .. code-tab:: toml +**Configuration fields:** - [src_trace.projects.project_name] - comment_type = "c" +- ``src_dir`` - Root directory for source file discovery (relative to Sphinx project root or the directory where the TOML config file is located if given) +- ``exclude`` - List of glob patterns to exclude from processing +- ``include`` - List of glob patterns to include (if empty, includes all files) +- ``gitignore`` - Whether to respect ``.gitignore`` rules when discovering files (Nested .gitignore is NOT supported yet) +- ``comment_type`` - Comment style for the programming language ("cpp" and "python" are currently supported) -.. _source_dir: +.. _`source_dir`: src_dir -~~~~~~~ +^^^^^^^ -The relative path from the ``conf.py`` or ``.toml`` file to the source code's root directory +Specifies the root directory for source file discovery. This path is resolved relative to the location of the TOML configuration file. -Default: **./** +**Type:** ``str`` +**Default:** ``"./"`` -.. tabs:: +.. code-block:: toml - .. code-tab:: python + [codelinks.projects.my_project.source_discover] + src_dir = "../src" - src_trace_projects = { - "project_name": { - "src_dir": "./../src" - } - } +**Examples:** - .. code-tab:: toml +- ``"./"`` - Current directory (relative to config file) +- ``"../src"`` - Parent directory's src folder +- ``"./my_project/source"`` - Subdirectory within current directory - [src_trace.projects.project_name] - src_dir = "./../src" +exclude +^^^^^^^ -remote_url_pattern -~~~~~~~~~~~~~~~~~~ +Defines a list of glob patterns for files and directories to exclude from discovery. This is useful for ignoring build artifacts, temporary files, or specific source files that shouldn't be processed. -This option only works with :ref:`src_trace_set_remote_url` set to **True**. -The pattern to access the source code to the remote repositories such as GitHub. +**Type:** ``list[str]`` +**Default:** ``[]`` -Default: **Not set** +.. code-block:: toml -.. tabs:: + [codelinks.projects.my_project.source_discover] + exclude = [ + "build/**" + "*.tmp" + "tests/fixtures/**" + "vendor/third_party/**" + ] - .. code-tab:: python +**Common exclusion patterns:** - src_trace_projects = { - "project_name": { - "remote_url_pattern": "https://github.com/useblocks/sphinx-codelinks/blob/{commit}/{path}#L{line}" - } - } +- ``"build/**"`` - Exclude entire build directory +- ``"*.o"`` - Exclude object files +- ``"**/__pycache__/**"`` - Exclude Python cache directories +- ``"node_modules/**"`` - Exclude Node.js dependencies - .. code-tab:: toml +include +^^^^^^^ - [src_trace.projects.project_name] - remote_url_pattern = "https://github.com/useblocks/sphinx-codelinks/blob/{commit}/{path}#L{line}" +Defines a list of glob patterns for files to explicitly include in discovery. When specified, only files matching these patterns will be processed, regardless of other filtering rules. -This option leverages the configuration of :external+needs:ref:`need_string_links` -with the following setup: +**Type:** ``list[str]`` +**Default:** ``[]`` (include all files) -.. code-block:: python +.. code-block:: toml - remote_url_pattern = remote_url_pattern.format( - commit=commit_id, - path=f"{remote_src_dir}/" + "{{value}}", - line="{{lineno}}", - ) + [codelinks.projects.my_project.source_discover] + include = [ + "src/**/*.cpp", + "src/**/*.h", + "include/**/*.hpp" + ] - { - "regex": r"^(?P.+)#L(?P.*)?", - "link_url": remote_url_pattern, - "link_name": "{{value}}#L{{lineno}}", - "options": [remote_url_field], - } +**Priority:** The ``include`` option has the highest priority and overrides both ``exclude`` and ``gitignore`` settings. -exclude -~~~~~~~ +**Common inclusion patterns:** + +- ``"**/*.cpp"`` - Include all C++ source files +- ``"**/*.py"`` - Include all Python files +- ``"src/**"`` - Include everything in src directory +- ``"*.{c,h}"`` - Include C source and header files -The option is a list of glob patterns to exclude the files which are not required to be addressed +comment_type +^^^^^^^^^^^^ -Default: **[]** +Specifies the comment syntax style used in the source code files. This determines what file types are discovered and how **Sphinx-CodeLinks** parses comments for documentation extraction. -.. tabs:: +**Type:** ``str`` +**Default:** ``"cpp"`` +**Supported values:** ``"cpp"``, ``"python"`` - .. code-tab:: python +.. code-block:: toml - src_trace_projects = { - "project_name": { - "exclude": ["dcdc/src/ubt/ubt.cpp"] - } - } + [codelinks.projects.my_project.source_discover] + comment_type = "python" - .. code-tab:: toml +**Supported comment styles:** - [src_trace.projects.project_name] - exclude = ["dcdc/src/ubt/ubt.cpp"] +.. list-table:: Title + :header-rows: 1 + :widths: 25, 25, 30, 50 -include + * - Language + - comment_type + - Comment Syntax + - discovered file types + * - C/C++ + - ``"cpp"`` + - ``//`` (single-line), + ``/* */`` (multi-line) + - ``c``, ``h``, ``.cpp``, and ``.hpp`` + * - Python + - ``"python"`` + - ``#`` (single-line), + ``""" """`` (docstrings) + - ``.py`` + +.. note:: Future versions may support additional programming languages. Currently, only C/C++ and Python comment styles are supported. + +gitignore +^^^^^^^^^ + +Controls whether to respect ``.gitignore`` files when discovering source files. When enabled, files and directories listed in ``.gitignore`` will be automatically excluded from processing. + +**Type:** ``bool`` +**Default:** ``true`` + +.. code-block:: toml + + [codelinks.projects.my_project.source_discover] + gitignore = false + +**Behavior:** + +- ``true`` - Respect ``.gitignore`` rules (recommended) +- ``false`` - Ignore ``.gitignore`` files and process all matching files + +.. important:: **Current Limitation:** This option only supports the root-level ``.gitignore`` file. Nested ``.gitignore`` files in subdirectories or parent directories are not currently processed. + +For more information about the usage examples, see :ref:`source discover `. + +.. _`analyse_config`: + +analyse ~~~~~~~ -The option is a list of glob patterns to include the files which are required to be addressed +Configures how **Sphinx-CodeLinks** analyse source files to extract markers from comments. This option defines how the markers in source code are parsed and extracted. -Default: **[]** +**Complete Configuration Example:** -.. tabs:: +.. code-block:: toml - .. code-tab:: python + [codelinks] + outdir = "output" - src_trace_projects = - { - "project_name": { - "include": ["dcdc/src/ubt/ubt.cpp"] - } - } + [codelinks.projects.my_project.source_discover] + src_dir = "./" + exclude = [] + include = [] + gitignore = true + comment_type = "cpp" - .. code-tab:: toml + [codelinks.projects.my_project.analyse] + get_need_id_refs = true + get_oneline_needs = true + get_rst = true - [src_trace.projects.project_name] - include = ["dcdc/src/ubt/ubt.cpp"] + [codelinks.projects.my_project.analyse.oneline_comment_style] + start_sequence = "@" + # End sequences is newline by default. Whether it is "\n" or "\r\n" depending on the platform + end_sequence = "\n" + field_split_char = "," + needs_fields = [ + { name = "title", type = "str" }, + { name = "id", type = "str" }, + { name = "type", type = "str", default = "impl" }, + { name = "links", type = "list[str]", default = [] }, + ] -.. note:: **include** option has the highest priority over **exclude** and **gitignore** options. + [codelinks.projects.my_project.analyse.need_id_refs] + markers = ["@need-ids:"] -gitignore -~~~~~~~~~ + [codelinks.projects.my_project.analyse.marked_rst] + start_sequence = "@rst" + end_sequence = "@endrst" + +get_need_id_refs +^^^^^^^^^^^^^^^^ + +Enables the extraction of need IDs from source code comments. When enabled, **SourceAnalyse** will parse comments for specific markers that indicate need IDs, allowing them to be extracted for further usages. + +**Type:** ``bool`` +**Default:** ``False`` + +.. code-block:: toml + + [codelinks.projects.my_project.analyse] + get_need_id_refs = true + +get_oneline_needs +^^^^^^^^^^^^^^^^^ -The option to respect the .gitignore file. +Enables the extraction of one-line needs directly from source code comments. When enabled, **SourceAnalyse** will parse comments for simplified :ref:`one-line patterns ` that represent needs, allowing them to be processed without requiring full RST syntax. -Default: **True** +**Type:** ``bool`` +**Default:** ``False`` -.. tabs:: +.. code-block:: toml - .. code-tab:: python + [codelinks.projects.my_project.analyse] + get_oneline_needs = false - src_trace_projects = { - "project_name": { - "gitignore": False - } +get_rst +^^^^^^^ - .. code-tab:: toml +Enables the extraction of marked RST text from source code comments. When enabled, **SourceAnalyse** will parse comments for specific markers that indicate RST blocks, allowing them to be extracted. - [src_trace.projects.project_name] - gitignore = false +**Type:** ``bool`` +**Default:** ``False`` -.. attention:: This option currently does NOT support nested .gitignore files +.. code-block:: toml + + [codelinks.projects.my_project.analyse] + get_rst = false .. _`oneline_comment_style`: -oneline_comment_style -~~~~~~~~~~~~~~~~~~~~~ +analyse.oneline_comment_style +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Enables the use of simplified :ref:`one-line comment patterns ` to represent **Sphinx-Needs** items directly in source code, eliminating the need for embedded RST syntax. + +**Type:** ``dict`` +**Default:** See below + +.. code-block:: toml + + [codelinks.projects.my_project.analyse.oneline_comment_style] + start_sequence = "@" + end_sequence = "\n" # Platform-specific line ending + field_split_char = "," + needs_fields = [ + { name = "title", type = "str" }, + { name = "id", type = "str" }, + { name = "type", type = "str", default = "impl" }, + { name = "links", type = "list[str]", default = [] }, + ] + +**Configuration fields:** + +- ``start_sequence`` - Character(s) that begin a one-line comment pattern +- ``end_sequence`` - Character(s) that end a one-line comment pattern (typically line ending) +- ``field_split_char`` - Character used to separate fields within the comment +- ``needs_fields`` - List of field definitions for extracting need information + +**Example usage:** + +The following one-line comment in source code: + +.. code-block:: cpp + + // @Function Bar, IMPL_4, impl, [SPEC_1, SPEC_2] + +Is equivalent to this RST directive: + +.. code-block:: rst + + .. impl:: Function Bar + :id: IMPL_4 + :links: SPEC_1, SPEC_2 -This option enables users to simply define a customized one-line-pattern comment to represent -``Sphinx-Needs`` need items instead of using RST. +.. important:: The ``type`` and ``title`` fields must be configured in ``needs_fields`` as they are mandatory for **Sphinx-Needs**. -Default: +analyse.need_id_refs +^^^^^^^^^^^^^^^^^^^^ -.. tabs:: +Configuration for Sphinx-Needs ID reference extraction. - .. code-tab:: python +**Type:** ``dict`` +**Default:** See below - import os - src_trace_projects = { - "project_name": { - "oneline_comment_style": { - "start_sequence": "@", - "end_sequence": os.linesep, - "field_split_char": ",", - needs_fields = [ - {"name": "title"}, - {"name": "id"}, - {"name": "type", "default": "impl"}, - {"name": "links", "type": "list[str]", "default": []}, - ] - } - } - } +.. code-block:: toml - .. code-tab:: toml + [codelinks.projects.my_project.analyse.need_id_refs] + markers = ["@need-ids:"] - [src_trace.projects.project_name.oneline_comment_style] - start_sequence = "@" - # end_sequence for the online comments; default is an os-dependant newline character - field_split_char = "," - needs_fields = [ - { "name" = "title", "type" = "str" }, - { "name" = "id", "type" = "str" }, - { "name" = "type", "type" = "str", "default" = "impl" }, - { "name" = "links", "type" = "list[str]", "default" = [] }, - ] +**Configuration fields:** -With the default, the following one-line comment will be extracted by ``CodeLinks`` and -it is equivalent to the following RST +- ``markers`` (``list[str]``) - List of marker strings that identify need ID references -.. tabs:: +analyse.marked_rst +^^^^^^^^^^^^^^^^^^ - .. code-tab:: c +Configuration for marked RST block extraction. - // @Function Bar, IMPL_4, impl, [SPEC_1, SPEC_2] +**Type:** ``dict`` +**Default:** See below - .. code-tab:: RST +.. code-block:: toml - .. impl:: Function Bar - :id: IMPL_4 - :links: [SPEC_1, SPEC_2] + [codelinks.projects.my_project.analyse.marked_rst] + start_sequence = "@rst" + end_sequence = "@endrst" -.. caution:: **type** and **title** must be configured in **needs_fields** as they are mandatory for Sphinx-Needs +**Configuration fields:** -More uses cases can be found in `tests `__ +- ``start_sequence`` (``str``) - Marker that begins an RST block +- ``end_sequence`` (``str``) - Marker that ends an RST block diff --git a/docs/source/components/directive.rst b/docs/source/components/directive.rst index 23aed28..4012aa5 100644 --- a/docs/source/components/directive.rst +++ b/docs/source/components/directive.rst @@ -3,7 +3,7 @@ Directive ========= -``CodeLinks`` provides ``src-trace`` directive and it can be used in the following ways: +``CodeLinks`` provides the ``src-trace`` directive and it can be used in the following ways: .. code-block:: rst @@ -19,17 +19,17 @@ or :project: project_config :directory: ./example -``src-trace`` directive has the following options: +The ``src-trace`` directive has the following options: -* **project**: the project config specified in ``conf.py`` or ``toml`` file to be used for source tracing. +* **project**: the project config specified in ``conf.py`` or TOML file to be used for source tracing. * **file**: the source file to be traced. * **directory**: the source files in the directory to be traced recursively. Regarding the **file** and **directory** options: -- they are optional and mutually exclusive. -- the given paths are relative to ``src_dir`` defined in the source tracing configuration -- if not given, the whole project will be examined. +- They are optional and mutually exclusive. +- The given paths are relative to ``src_dir`` defined in the source tracing configuration. +- If not given, the whole project will be examined. Example ------- @@ -44,9 +44,8 @@ With the following configuration for a demo source code project `dcdc `. diff --git a/docs/source/components/discover.rst b/docs/source/components/discover.rst new file mode 100644 index 0000000..db31c72 --- /dev/null +++ b/docs/source/components/discover.rst @@ -0,0 +1,40 @@ +.. _discover: + +Source Discover +=============== + +SourceDiscover is one of the modules provided in ``Codelinks``. It discovers the source files from the given directory. +It provides users CLI and API to discover the source files. + + +Usage Examples +-------------- + +**Basic Configuration:** + +.. code-block:: toml + + [source_discover] + src_dir = "./src" + comment_type = "cpp" + +**Advanced Filtering:** + +.. code-block:: toml + + [source_discover] + src_dir = "./" + include = [] + exclude = ["src/legacy/**", "**/*_test.cpp"] + gitignore = true + comment_type = "cpp" + +**Python Project:** + +.. code-block:: toml + + [source_discover] + src_dir = "./my_package" + include = [] + exclude = ["tests/**", "setup.py"] + comment_type = "python" diff --git a/docs/source/components/oneline.rst b/docs/source/components/oneline.rst index b810391..63ee49b 100644 --- a/docs/source/components/oneline.rst +++ b/docs/source/components/oneline.rst @@ -9,10 +9,15 @@ to simplify the effort required to create a need in source code. :ref:`Here ` is the default one-line comment style. +**Additional examples and use cases:** + +For more comprehensive examples and advanced configurations, see the `test cases `__. + + Start and End sequences ----------------------- -To have a better understanding of the syntax of a one-line comment, we will break it down into the following: +To have a better understanding of the syntax of a one-line comment, we will break it down as follows: **start_sequence** defines the characters where the one-line comment starts. **end_sequence** defines the characters where the one-line comment ends. @@ -95,13 +100,13 @@ For example, with the following **needs_fields** configuration: {"name": "links", "type": "list[str]", "default": []}, ], -the one-line comment shall be defined as the following +the one-line comment shall be defined as follows: .. tabs:: .. code-tab:: c - // @ title, id_123, implementation, [link1, link2] + // @ title, id_123, implementation, [link1, link2] .. code-tab:: rst @@ -149,14 +154,14 @@ This means the ``order of needs_fields`` determines ``the position of the field` For example, with the mentioned :ref:`needs_fields definition ` -field ``title`` is the first element is the list, so the string of the title must be -the first field in the one-line comment +field ``title`` is the first element in the list, so the string of the title must be +the first field in the one-line comment. .. tabs:: .. code-tab:: c - // @ this is title, this is id, this_type, [link1, link2] + // @ this is title, this is id, this_type, [link1, link2] .. code-tab:: rst diff --git a/docs/source/development/contributing.rst b/docs/source/development/contributing.rst index 03b3128..4ca31a9 100644 --- a/docs/source/development/contributing.rst +++ b/docs/source/development/contributing.rst @@ -6,11 +6,11 @@ This page provides a guide for developers wishing to contribute to ``Sphinx-Code Bugs, Features and PRs ---------------------- -For **bug reports** and well-described **technical feature request**, please use our issue tracker: +For **bug reports** and well-described **technical feature requests**, please use our issue tracker: https://github.com/useblocks/sphinx-codelinks/issues -If you have already created a PR, you can send it in. Our CI workflow will check (test and code styles) -and a maintainer will perform a review, before we can merge it. +If you have already created a PR, you can send it in. Our CI workflow will check (tests and code styles) +and a maintainer will perform a review before we can merge it. Your PR should conform with the following rules: - A meaningful description or link, which describes the change @@ -24,7 +24,7 @@ Install Dependencies ``CodeLinks`` uses `rye `_ to manage the repository. -For the development, use the following command to install python dependencies into the virtual environment. +For development, use the following command to install Python dependencies into the virtual environment. .. code-block:: bash @@ -40,7 +40,7 @@ To run the formatting and linting, pre-commit is used: pre-commit install # to auto-run on every commit pre-commit run --all-files # to run manually -The CI also checks typing, use the following command locally to see if your code is well-typed +The CI also checks typing. Use the following command locally to see if your code is well-typed: .. code-block:: bash diff --git a/docs/source/index.rst b/docs/source/index.rst index 1fe53dd..f1bcc25 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -62,6 +62,7 @@ Contents components/directive components/oneline components/analyse + components/discover .. toctree:: :maxdepth: 1 diff --git a/docs/src_trace.toml b/docs/src_trace.toml index 304299d..cc7a6a3 100644 --- a/docs/src_trace.toml +++ b/docs/src_trace.toml @@ -1,16 +1,22 @@ -[src_trace] +[codelinks] # Configuration for source tracing set_local_url = true # Set to true to enable local code html and URL generation local_url_field = "local-url" # Need's field name for local URL set_remote_url = true # Set to true to enable remote url to be generated remote_url_field = "remote-url" # Need's field name for remote URL -[src_trace.projects.dcdc] # Configuration for source tracing project "dcdc" -src_dir = "../tests/data/dcdc" # Relative path from conf.py to the source directory +[codelinks.projects.dcdc] remote_url_pattern = "https://github.com/useblocks/sphinx-codelinks/blob/{commit}/{path}#L{line}" # URL pattern for remote source code -[src_trace.projects.dcdc.oneline_comment_style] +[codelinks.projects.dcdc.source_discover] +src_dir = "../tests/data/dcdc" # Relative path from this TOML config to the source directory + +[codelinks.projects.dcdc.analyse] +get_need_id_refs = false +get_oneline_needs = true + +[codelinks.projects.dcdc.analyse.oneline_comment_style] # Configuration for oneline comment style start_sequence = "[[" # Start sequence for oneline comments end_sequence = "]]" # End sequence for the online comments; default is newline character @@ -24,6 +30,8 @@ needs_fields = [ ] }, ] -[src_trace.projects.src] -src_dir = "../tests/doc_test/minimum_config" +[codelinks.projects.src] remote_url_pattern = "https://github.com/useblocks/sphinx-codelinks/blob/{commit}/{path}#L{line}" + +[codelinks.projects.src.source_discover] +src_dir = "../tests/doc_test/minimum_config" diff --git a/src/sphinx_codelinks/analyse/analyse.py b/src/sphinx_codelinks/analyse/analyse.py index 40cc273..b100118 100644 --- a/src/sphinx_codelinks/analyse/analyse.py +++ b/src/sphinx_codelinks/analyse/analyse.py @@ -3,16 +3,11 @@ import json import logging from pathlib import Path -from typing import Any +from typing import Any, TypedDict from tree_sitter import Node as TreeSitterNode from sphinx_codelinks.analyse import utils -from sphinx_codelinks.analyse.config import ( - UNIX_NEWLINE, - OneLineCommentStyle, - SourceAnalyseConfig, -) from sphinx_codelinks.analyse.models import ( MarkedContentType, MarkedRst, @@ -26,6 +21,11 @@ OnelineParserInvalidWarning, oneline_parser, ) +from sphinx_codelinks.config import ( + UNIX_NEWLINE, + OneLineCommentStyle, + SourceAnalyseConfig, +) # initialize logger logger = logging.getLogger(__name__) @@ -36,6 +36,14 @@ logger.addHandler(console) +class AnalyseWarningType(TypedDict): + file_path: str + lineno: int + msg: str + type: str + sub_type: str + + @dataclass class AnalyseWarning: file_path: str @@ -46,8 +54,6 @@ class AnalyseWarning: class SourceAnalyse: - warning_filepath: Path = Path("cached_warnings") / "codelinks_warnings.json" - def __init__( self, analyse_config: SourceAnalyseConfig, @@ -70,7 +76,6 @@ def __init__( self.git_root if self.git_root else self.analyse_config.src_dir ) self.oneline_warnings: list[AnalyseWarning] = [] - self.warnings_path = analyse_config.outdir / SourceAnalyse.warning_filepath def get_src_strings(self) -> Generator[tuple[Path, bytes], Any, None]: # type: ignore[explicit-any] """Load source files and extract their content.""" @@ -357,8 +362,8 @@ def merge_marked_content(self) -> None: key=lambda x: (x.filepath, x.source_map["start"]["row"]) ) - def dump_marked_content(self) -> None: - output_path = self.analyse_config.outdir / "marked_content.json" + def dump_marked_content(self, outdir: Path) -> None: + output_path = outdir / "marked_content.json" if not output_path.parent.exists(): output_path.parent.mkdir(parents=True) to_dump = [ @@ -372,51 +377,3 @@ def run(self) -> None: self.create_src_objects() self.extract_marked_content() self.merge_marked_content() - self.dump_marked_content() - self.update_warnings() - - @classmethod - def load_warnings(cls, warnings_dir: Path) -> list[AnalyseWarning] | None: - """Load warnings from the given path. - - It mainly used for other apps or users to load warnings files directly. - """ - warnings_path = warnings_dir / cls.warning_filepath - if not warnings_path.exists(): - return None - with warnings_path.open("r") as f: - # load the json file and convert to AnalyseWarning] - warnings = json.load(f) - loaded_warnings = [AnalyseWarning(**warning) for warning in warnings] - return loaded_warnings - - def _load_warnings(self) -> list[AnalyseWarning] | None: - if not self.warnings_path.exists(): - return None - with self.warnings_path.open("r") as f: - # load the json file and convert to AnalyseWarning] - warnings = json.load(f) - loaded_warnings = [AnalyseWarning(**warning) for warning in warnings] - return loaded_warnings - - def update_warnings(self) -> None: - loaded_warnings = self._load_warnings() - current_warnings = [_warning.__dict__ for _warning in self.oneline_warnings] - if loaded_warnings: - _warnings = [_warning.__dict__ for _warning in loaded_warnings] - cached_warnings = [ - _warning - for _warning in _warnings - if _warning["file_path"] - not in [str(src_file) for src_file in self.src_files] - ] - total_warning = cached_warnings + current_warnings - else: - total_warning = current_warnings - if not self.warnings_path.parent.exists(): - self.warnings_path.parent.mkdir(parents=True) - with self.warnings_path.open("w") as f: - json.dump( - total_warning, - f, - ) diff --git a/src/sphinx_codelinks/analyse/oneline_parser.py b/src/sphinx_codelinks/analyse/oneline_parser.py index 5ac1b16..ff76da0 100644 --- a/src/sphinx_codelinks/analyse/oneline_parser.py +++ b/src/sphinx_codelinks/analyse/oneline_parser.py @@ -2,7 +2,7 @@ from enum import Enum import logging -from sphinx_codelinks.analyse.config import ( +from sphinx_codelinks.config import ( ESCAPE, SUPPORTED_COMMENT_TYPES, UNIX_NEWLINE, diff --git a/src/sphinx_codelinks/analyse/projects.py b/src/sphinx_codelinks/analyse/projects.py new file mode 100644 index 0000000..4a1d873 --- /dev/null +++ b/src/sphinx_codelinks/analyse/projects.py @@ -0,0 +1,84 @@ +import json +import logging +from pathlib import Path +from typing import cast + +from sphinx_codelinks.analyse.analyse import ( + AnalyseWarning, + AnalyseWarningType, + SourceAnalyse, +) +from sphinx_codelinks.config import CodeLinksConfig, CodeLinksProjectConfigType + +# initialize logger +logger = logging.getLogger(__name__) +logger.setLevel(logging.INFO) +# log to the console +console = logging.StreamHandler() +console.setLevel(logging.INFO) +logger.addHandler(console) + + +class AnalyseProjects: + warning_filepath: Path = Path("warnings") / "codelinks_warnings.json" + + def __init__(self, codelink_config: CodeLinksConfig) -> None: + self.projects_configs: dict[str, CodeLinksProjectConfigType] = ( + codelink_config.projects + ) + self.projects_analyse: dict[str, SourceAnalyse] = {} + self.warnings_path = codelink_config.outdir / AnalyseProjects.warning_filepath + self.outdir = codelink_config.outdir + + def run(self) -> None: + for project, config in self.projects_configs.items(): + src_analyse = SourceAnalyse(config["analyse_config"]) + src_analyse.run() + self.projects_analyse[project] = src_analyse + + def dump_markers(self) -> None: + output_path = self.outdir / "marked_content.json" + if not output_path.parent.exists(): + output_path.parent.mkdir(parents=True) + to_dump = { + project: [marker.to_dict() for marker in analyse.all_marked_content] + for project, analyse in self.projects_analyse.items() + } + with output_path.open("w") as f: + json.dump(to_dump, f) + logger.info(f"Marked content dumped to {output_path}") + + @classmethod + def load_warnings(cls, warnings_dir: Path) -> list[AnalyseWarning] | None: + """Load warnings from the given path. + + It mainly used for other apps or users to load warnings files directly. + """ + warnings_path = warnings_dir / cls.warning_filepath + if not warnings_path.exists(): + return None + with warnings_path.open("r") as f: + # load the json file and convert to AnalyseWarning] + warnings = json.load(f) + loaded_warnings: list[AnalyseWarning] = [ + AnalyseWarning(**warning) for warning in warnings + ] + + return loaded_warnings + + def update_warnings(self) -> None: + current_warnings: list[AnalyseWarningType] = [ + cast(AnalyseWarningType, _warning.__dict__) + for analyse in self.projects_analyse.values() + for _warning in analyse.oneline_warnings + ] + self.dump_warnings(current_warnings) + + def dump_warnings(self, warnings: list[AnalyseWarningType]) -> None: + if not self.warnings_path.parent.exists(): + self.warnings_path.parent.mkdir(parents=True) + with self.warnings_path.open("w") as f: + json.dump( + warnings, + f, + ) diff --git a/src/sphinx_codelinks/analyse/utils.py b/src/sphinx_codelinks/analyse/utils.py index 925f86d..027abc0 100644 --- a/src/sphinx_codelinks/analyse/utils.py +++ b/src/sphinx_codelinks/analyse/utils.py @@ -9,7 +9,8 @@ from tree_sitter import Language, Parser, Point, Query, QueryCursor from tree_sitter import Node as TreeSitterNode -from sphinx_codelinks.analyse.config import UNIX_NEWLINE, CommentCategory, CommentType +from sphinx_codelinks.config import UNIX_NEWLINE, CommentCategory +from sphinx_codelinks.source_discover.config import CommentType # initialize logger logger = logging.getLogger(__name__) @@ -133,6 +134,7 @@ def locate_git_root(src_dir: Path) -> Path | None: for parent in parents: if (parent / ".git").exists() and (parent / ".git").is_dir(): return parent + logger.warning(f"git root is not found in the parent of {src_dir}") return None diff --git a/src/sphinx_codelinks/cmd.py b/src/sphinx_codelinks/cmd.py index 1a65422..53dde99 100644 --- a/src/sphinx_codelinks/cmd.py +++ b/src/sphinx_codelinks/cmd.py @@ -1,6 +1,4 @@ from collections import deque -from collections.abc import Callable -from enum import Enum from os import linesep from pathlib import Path import tomllib @@ -8,26 +6,19 @@ import typer -from sphinx_codelinks.analyse.analyse import SourceAnalyse -from sphinx_codelinks.analyse.config import ( - COMMENT_FILETYPE, - MarkedRstConfig, - MarkedRstConfigType, - NeedIdRefsConfig, - NeedIdRefsConfigType, - OneLineCommentStyle, - OneLineCommentStyleType, - SourceAnalyseConfig, - SourceAnalyseConfigFileType, - SourceAnalyseConfigType, - SrcDiscoverConfigType4Analyse, +from sphinx_codelinks.analyse.projects import AnalyseProjects +from sphinx_codelinks.config import ( + CodeLinksConfig, + CodeLinksConfigType, + CodeLinksProjectConfigType, + generate_project_configs, ) from sphinx_codelinks.source_discover.config import ( + CommentType, SourceDiscoverConfig, SourceDiscoverConfigType, ) from sphinx_codelinks.source_discover.source_discover import SourceDiscover -from sphinx_codelinks.sphinx_extension.config import SrcTraceProjectConfigFileType app = typer.Typer( no_args_is_help=True, context_settings={"help_option_names": ["-h", "--help"]} @@ -46,6 +37,15 @@ def analyse( exists=True, ), ], + projects: Annotated[ + list[str] | None, + typer.Option( + "--project", + "-p", + help="Specify the project name of the config. If not specified, take all", + show_default=True, + ), + ] = None, outdir: Annotated[ Path | None, typer.Option( @@ -60,92 +60,67 @@ def analyse( ] = None, ) -> None: """Analyse marked content in source code.""" - data = load_src_analyse_config_from_toml(config) - errors: deque[str] = deque() - - # Get source_discover configuration - src_discover_4_analyse_dict = cast( - SrcDiscoverConfigType4Analyse | None, data.get("source_discover") - ) - src_discover_config = cast( - SourceDiscoverConfig, - convert_dict_2_config(src_discover_4_analyse_dict, ConfigType.SourceDiscover), - ) - - src_discover_errors = src_discover_config.check_schema() - - if src_discover_errors: - errors.appendleft("Invalid source discovery configuration:") - errors.extend(src_discover_errors) - if errors: - raise typer.BadParameter(f"{linesep.join(errors)}") + data: CodeLinksConfigType = load_config_from_toml(config) - # src dir shall be relevant to the config file's location - src_discover_config.src_dir = ( - config.parent / src_discover_config.src_dir - ).resolve() - - src_discover = SourceDiscover(src_discover_config) - # Get oneline_comment_style configuration - oneline_comment_style_dict: OneLineCommentStyleType | None = data.get( - "oneline_comment_style" - ) - oneline_comment_style: OneLineCommentStyle = cast( - OneLineCommentStyle, - convert_dict_2_config( - oneline_comment_style_dict, ConfigType.OneLineCommentStyle - ), - ) - - oneline_errors = oneline_comment_style.check_fields_configuration() - if oneline_errors: - errors.appendleft("Invalid oneline comment style configuration:") - errors.extend(oneline_errors) - - # Get need_id_refs configuration - need_id_refs_config_dict: NeedIdRefsConfigType | None = data.get("need_id_refs") - need_id_refs_config = cast( - NeedIdRefsConfig, - convert_dict_2_config(need_id_refs_config_dict, ConfigType.NeedIdRefsConfig), - ) - - # Get marked_rst configuration - marked_rst_config_dict: MarkedRstConfigType | None = data.get("marked_rst") - marked_rst_config = cast( - MarkedRstConfig, - convert_dict_2_config(marked_rst_config_dict, ConfigType.MarkedRstCofig), - ) - - non_root_configs = { - "source_discover", - "oneline_comment_style", - "need_id_refs", - "marked_rst", - } + try: + codelinks_config = CodeLinksConfig(**data) + generate_project_configs(codelinks_config.projects) + except TypeError as e: + raise typer.BadParameter(str(e)) from e - # Get root config - src_analyse_dict: SourceAnalyseConfigType = cast( - SourceAnalyseConfigType, - {key: value for key, value in data.items() if key not in non_root_configs}, - ) - src_analyse_dict["src_files"] = src_discover.source_paths - src_analyse_dict["src_dir"] = Path(src_discover.src_discover_config.src_dir) - src_analyse_dict["need_id_refs_config"] = need_id_refs_config - src_analyse_dict["marked_rst_config"] = marked_rst_config - src_analyse_dict["oneline_comment_style"] = oneline_comment_style + errors: deque[str] = deque() if outdir: - src_analyse_dict["outdir"] = outdir - - src_analyse_config = SourceAnalyseConfig(**src_analyse_dict) - - analyse_errors = src_analyse_config.check_fields_configuration() - errors.extend(analyse_errors) - if errors: - raise typer.BadParameter(f"{linesep.join(errors)}") - - src_analyse = SourceAnalyse(src_analyse_config) - src_analyse.run() + codelinks_config.outdir = outdir + + project_errors: list[str] = [] + if projects: + for project in projects: + if project not in codelinks_config.projects: + if not project_errors: + project_errors.append("The following projects are not found:") + project_errors.append(project) + if project_errors: + raise typer.BadParameter(f"{linesep.join(project_errors)}") + + specifed_project_configs: dict[str, CodeLinksProjectConfigType] = {} + for project, _config in codelinks_config.projects.items(): + if projects and project not in projects: + continue + # Get source_discover configuration + src_discover_config = _config["source_discover_config"] + + src_discover_errors = src_discover_config.check_schema() + + if src_discover_errors: + errors.appendleft("Invalid source discovery configuration:") + errors.extend(src_discover_errors) + if errors: + raise typer.BadParameter(f"{linesep.join(errors)}") + + # src dir shall be relevant to the config file's location + src_discover_config.src_dir = ( + config.parent / src_discover_config.src_dir + ).resolve() + + src_discover = SourceDiscover(src_discover_config) + + # Init source analyse config + analyse_config = _config["analyse_config"] + analyse_config.src_files = src_discover.source_paths + analyse_config.src_dir = Path(src_discover.src_discover_config.src_dir) + + analyse_errors = analyse_config.check_fields_configuration() + errors.extend(analyse_errors) + if errors: + raise typer.BadParameter(f"{linesep.join(errors)}") + + specifed_project_configs[project] = {"analyse_config": analyse_config} + + codelinks_config.projects = specifed_project_configs + analyse_projects = AnalyseProjects(codelinks_config) + analyse_projects.run() + analyse_projects.dump_markers() @app.command(no_args_is_help=True) @@ -163,43 +138,46 @@ def discover( ), ], exclude: Annotated[ - list[str] | None, + list[str], typer.Option( "--excludes", "-e", help="Glob patterns to be excluded.", ), - ] = None, + ] = [], # noqa: B006 # to show the default value on CLI include: Annotated[ - list[str] | None, + list[str], typer.Option( "--includes", "-i", help="Glob patterns to be included.", ), - ] = None, - gitignore: Annotated[bool, typer.Option(help="Respect .gitignore(s)")] = True, - file_types: Annotated[ - list[str] | None, + ] = [], # noqa: B006 # to show the default value on CLI + gitignore: Annotated[ + bool, typer.Option( - "--file-type", - "-f", - help="The file extension to be discovered. If not specified, all files are discovered.", + help="Respect .gitignore in the given directory. Nested .gitignore Not supported" ), - ] = None, + ] = True, + comment_type: Annotated[ + CommentType, + typer.Option( + "--comment-type", + "-c", + help="The relevant file extensions which use the specified the comment type will be discovered.", + ), + ] = CommentType.cpp, ) -> None: """Discover the filepaths from the given root directory.""" src_discover_dict: SourceDiscoverConfigType = { "src_dir": src_dir, - "exclude": exclude or [], - "include": include or [], + "exclude": exclude, + "include": include, "gitignore": gitignore, + "comment_type": comment_type, } - if file_types is not None: - src_discover_dict["file_types"] = file_types - src_discover_config = SourceDiscoverConfig(**src_discover_dict) errors = src_discover_config.check_schema() @@ -212,148 +190,22 @@ def discover( typer.echo(file_path) -def load_config_from_toml( - toml_file: Path, project: str | None = None -) -> SrcTraceProjectConfigFileType: +def load_config_from_toml(toml_file: Path) -> CodeLinksConfigType: try: with toml_file.open("rb") as f: toml_data = tomllib.load(f) - if project: - toml_data = toml_data["src_trace"]["projects"][project] - except Exception as e: - raise Exception( - f"Failed to load source tracing configuration from {toml_file}" + raise typer.BadParameter( + f"Failed to load CodeLinks configuration from {toml_file}" ) from e - return cast(SrcTraceProjectConfigFileType, toml_data) + codelink_dict = toml_data.get("codelinks") + if not codelink_dict: + raise typer.BadParameter(f"No 'codelinks' section found in {toml_file}") -def load_src_analyse_config_from_toml(toml_file: Path) -> SourceAnalyseConfigFileType: - try: - with toml_file.open("rb") as f: - toml_data = tomllib.load(f) - - except Exception as e: - raise Exception( - f"Failed to load Source analyse configuration from {toml_file}" - ) from e - - return cast(SourceAnalyseConfigFileType, toml_data) - - -class ConfigType(str, Enum): - SourceDiscover = "source_discover" - OneLineCommentStyle = "oneline_comment_style" - NeedIdRefsConfig = "need_id_refs" - MarkedRstCofig = "marked_rst" - AnalzerConfig = "source_analyse" - - -def convert_dict_2_config( - config_dict: SrcDiscoverConfigType4Analyse - | OneLineCommentStyleType - | NeedIdRefsConfigType - | MarkedRstConfigType - | None, - config_type: ConfigType, -) -> SourceDiscoverConfig | OneLineCommentStyle | NeedIdRefsConfig | MarkedRstConfig: - func_map: dict[ - ConfigType, - Callable[ - [ - SrcDiscoverConfigType4Analyse - | OneLineCommentStyleType - | NeedIdRefsConfigType - | MarkedRstConfigType - | None - ], - SourceDiscoverConfig - | OneLineCommentStyle - | NeedIdRefsConfig - | MarkedRstConfig, - ], - ] = { - ConfigType.SourceDiscover: convert_src_discovery_config, # type: ignore[dict-item] # the type is restrict by its key already - ConfigType.OneLineCommentStyle: convert_oneline_comment_style_config, # type: ignore[dict-item] # the type is restrict by its key already - ConfigType.NeedIdRefsConfig: convert_need_id_refs_config, # type: ignore[dict-item] # the type is restrict by its key already - ConfigType.MarkedRstCofig: convert_marked_rst_config, # type: ignore[dict-item] # the type is restrict by its key already - } - - config = func_map[config_type](config_dict) - - return config - - -def convert_src_discovery_config( - config_dict: SrcDiscoverConfigType4Analyse | None, -) -> SourceDiscoverConfig: - if config_dict: - src_dicover_dict = {} - for key, value in config_dict.items(): - if key == "comment_type" and isinstance(value, list): - file_types = [] - for comment_type in value: - file_types.extend(COMMENT_FILETYPE[comment_type]) - src_dicover_dict["file_types"] = file_types - else: - src_dicover_dict[key] = value # type: ignore[assignment] # dynamic assignment - src_dicover_dict["src_dir"] = ( - Path(src_dicover_dict["src_dir"]) - if isinstance(src_dicover_dict["src_dir"], str) - else src_dicover_dict["src_dir"] - ) - src_discover_config = SourceDiscoverConfig(**src_dicover_dict) # type: ignore[arg-type] # mypy is confused by dynamic assignment - else: - src_discover_config = SourceDiscoverConfig() - - return src_discover_config - - -def convert_oneline_comment_style_config( - config_dict: OneLineCommentStyleType | None, -) -> OneLineCommentStyle: - if config_dict is None: - oneline_comment_style = OneLineCommentStyle() - else: - try: - oneline_comment_style = OneLineCommentStyle(**config_dict) - except TypeError as e: - raise typer.BadParameter( - f"Invalid oneline comment style configuration: {e}" - ) from e - return oneline_comment_style - - -def convert_need_id_refs_config( - config_dict: NeedIdRefsConfigType | None, -) -> NeedIdRefsConfig: - if not config_dict: - need_id_refs_config = NeedIdRefsConfig() - else: - try: - need_id_refs_config = NeedIdRefsConfig(**config_dict) - except TypeError as e: - raise typer.BadParameter( - f"Invalid oneline comment style configuration: {e}" - ) from e - return need_id_refs_config - - -def convert_marked_rst_config( - config_dict: MarkedRstConfigType | None, -) -> MarkedRstConfig: - if not config_dict: - marked_rst_config = MarkedRstConfig() - else: - try: - marked_rst_config = MarkedRstConfig(**config_dict) - except TypeError as e: - raise typer.BadParameter( - f"Invalid oneline comment style configuration: {e}" - ) from e - return marked_rst_config + return cast(CodeLinksConfigType, codelink_dict) if __name__ == "__main__": diff --git a/src/sphinx_codelinks/analyse/config.py b/src/sphinx_codelinks/config.py similarity index 52% rename from src/sphinx_codelinks/analyse/config.py rename to src/sphinx_codelinks/config.py index caad889..6745f86 100644 --- a/src/sphinx_codelinks/analyse/config.py +++ b/src/sphinx_codelinks/config.py @@ -5,16 +5,19 @@ from typing import Any, Literal, TypedDict, cast from jsonschema import ValidationError, validate +from sphinx.application import Sphinx +from sphinx.config import Config as _SphinxConfig -UNIX_NEWLINE = "\n" - +from sphinx_codelinks.source_discover.config import ( + CommentType, + SourceDiscoverConfig, + SourceDiscoverSectionConfigType, +) +from sphinx_codelinks.source_discover.source_discover import SourceDiscover -class CommentType(str, Enum): - python = "python" - cpp = "cpp" +UNIX_NEWLINE = "\n" -COMMENT_FILETYPE = {"cpp": ["c", "cpp", "h", "hpp"], "python": ["py"]} COMMENT_MARKERS = {CommentType.cpp: ["//", "/*"], CommentType.python: ["#"]} ESCAPE = "\\" SUPPORTED_COMMENT_TYPES = {"c", "h", "cpp", "hpp", "py"} @@ -25,14 +28,6 @@ class CommentCategory(str, Enum): docstring = "expression_statement" -class SrcDiscoverConfigType4Analyse(TypedDict, total=False): - src_dir: str - exclude: list[str] - include: list[str] - gitignore: bool - comment_type: list[CommentType] - - class NeedIdRefsConfigType(TypedDict): markers: list[str] @@ -298,8 +293,9 @@ def get_pos_list_str(self) -> list[int]: return pos_list_str -class SourceAnalyseConfigFileType(TypedDict, total=False): - src_discover: SrcDiscoverConfigType4Analyse +class AnalyseSectionConfigType(TypedDict, total=False): + """Define typing for loading `analyse` section from the file.""" + get_need_id_refs: bool get_oneline_needs: bool get_rst: bool @@ -310,9 +306,10 @@ class SourceAnalyseConfigFileType(TypedDict, total=False): class SourceAnalyseConfigType(TypedDict, total=False): + """Define typing for its API configuration.""" + src_files: list[Path] src_dir: Path - outdir: Path comment_type: CommentType get_need_id_refs: bool get_oneline_needs: bool @@ -322,6 +319,10 @@ class SourceAnalyseConfigType(TypedDict, total=False): oneline_comment_style: OneLineCommentStyle +class ProjectsAnalyseConfigType(TypedDict, total=False): + projects_config: dict[str, SourceAnalyseConfigType] + + @dataclass class SourceAnalyseConfig: @classmethod @@ -329,6 +330,7 @@ def field_names(cls) -> set[str]: return {item.name for item in fields(cls)} src_files: list[Path] = field( + default_factory=list, metadata={"schema": {"type": "array", "items": {"type": "string"}}}, ) """A list of source files to be processed.""" @@ -336,11 +338,6 @@ def field_names(cls) -> set[str]: default_factory=lambda: Path("./"), metadata={"schema": {"type": "string"}} ) - outdir: Path = field( - default=Path("output"), metadata={"schema": {"type": "string"}} - ) - """The directory where the virtual documents and their caches will be stored.""" - comment_type: CommentType = field( default=CommentType.cpp, metadata={"schema": {"type": "string"}} ) @@ -447,3 +444,413 @@ def check_fields_configuration(self) -> list[str]: errors.appendleft("analyse configuration errors:") errors.extend(analyse_errors) return list(errors) + + +SRC_TRACE_CACHE: str = "src_trace_cache" + + +class SourceTracingLineHref: + """Global class for the mapping between source file line numbers and Sphinx documentation links.""" + + def __init__(self) -> None: + self.mappings: dict[str, dict[int, str]] = {} + + +file_lineno_href = SourceTracingLineHref() + + +class CodeLinksProjectConfigType(TypedDict, total=False): + """TypedDict defining the configuration structure for individual SrcTrace projects. + + Contains both user-provided configuration: + - source_discover + - remote_url_pattern + - analyse + and runtime-generated configuration objects + - source_discover_config + - analyse_config + """ + + source_discover: SourceDiscoverSectionConfigType + remote_url_pattern: str + analyse: AnalyseSectionConfigType + source_discover_config: SourceDiscoverConfig + analyse_config: SourceAnalyseConfig + + +class CodeLinksConfigType(TypedDict): + config_from_toml: str | None + set_local_url: bool + local_url_field: str + set_remote_url: bool + remote_url_field: str + outdir: Path + projects: dict[str, CodeLinksProjectConfigType] + debug_measurement: bool + debug_filters: bool + + +@dataclass +class CodeLinksConfig: + @classmethod + def from_sphinx(cls, sphinx_config: _SphinxConfig) -> "CodeLinksConfig": + obj = cls() + super().__setattr__(obj, "_sphinx_config", sphinx_config) + return obj + + def __getattribute__(self, name: str) -> Any: # type: ignore[explicit-any] + if name.startswith("__") or name == "_sphinx_config": + return super().__getattribute__(name) + sphinx_config = ( + object.__getattribute__(self, "_sphinx_config") + if "_sphinx_config" in self.__dict__ + else None + ) + if sphinx_config: + return getattr( + super().__getattribute__("_sphinx_config"), f"src_trace_{name}" + ) + + return object.__getattribute__(self, name) + + def __setattr__(self, name: str, value: Any) -> None: # type: ignore[explicit-any] + if name == "_sphinx_config" and "src_trace_projects" in value: + src_trace_projects: dict[str, CodeLinksProjectConfigType] = value[ + "src_trace_projects" + ] + generate_project_configs(src_trace_projects) + + if name.startswith("__") or name == "_sphinx_config": + return super().__setattr__(name, value) + + sphinx_config = ( + object.__getattribute__(self, "_sphinx_config") + if "_sphinx_config" in self.__dict__ + else None + ) + + if sphinx_config: + setattr( + super().__getattribute__("_sphinx_config"), f"src_trace_{name}", value + ) + + if name == "outdir" and isinstance(value, str): + # Ensure outdir is a Path object + value = Path(value) + return object.__setattr__(self, name, value) + + @classmethod + def add_config_values(cls, app: Sphinx) -> None: + """Add all config values to Sphinx application""" + for item in fields(cls): + if item.default_factory is not MISSING: + default = item.default_factory() + elif item.default is not MISSING: + default = item.default + else: + raise Exception(f"Field {item.name} has no default value or factory") + + name = item.name + app.add_config_value( + f"src_trace_{name}", + default, + item.metadata["rebuild"], + types=item.metadata["types"], + ) + + @classmethod + def field_names(cls) -> set[str]: + return {item.name for item in fields(cls)} + + @classmethod + def get_schema(cls, name: str) -> dict[str, Any] | None: # type: ignore[explicit-any] + """Get the schema for a config item.""" + _field = next(field for field in fields(cls) if field.name is name) + if _field.metadata is not MISSING and "schema" in _field.metadata: + return _field.metadata["schema"] # type: ignore[no-any-return] + return None + + config_from_toml: str | None = field( + default=None, + metadata={ + "rebuild": "env", + "types": (str, type(None)), + "schema": { + "type": ["string", "null"], + "examples": ["config.toml", None], + }, + }, + ) + """Path to a TOML file to load configuration from.""" + + set_local_url: bool = field( + default=False, + metadata={ + "rebuild": "env", + "types": (bool,), + "schema": { + "type": "boolean", + }, + }, + ) + """Set the file URL in the extracted need.""" + + local_url_field: str = field( + default="local-url", + metadata={ + "rebuild": "env", + "types": (str,), + "schema": { + "type": "string", + }, + }, + ) + """The field name for the file URL in the extracted need.""" + + set_remote_url: bool = field( + default=False, + metadata={ + "rebuild": "env", + "types": (bool,), + "schema": { + "type": "boolean", + }, + }, + ) + remote_url_field: str = field( + default="remote-url", + metadata={ + "rebuild": "env", + "types": (str,), + "schema": { + "type": "string", + }, + }, + ) + """The field name for the remote URL in the extracted need.""" + + outdir: Path = field( + default=Path("output"), + metadata={"rebuild": "env", "types": (str), "schema": {"type": "string"}}, + ) + """The directory where the generated artifacts and their caches will be stored.""" + + projects: dict[str, CodeLinksProjectConfigType] = field( + default_factory=dict, + metadata={ + "rebuild": "env", + "types": (), + "schema": { + "type": "object", + "additionalProperties": { + "type": "object", + "properties": { + "source_discover": {}, + "analyse": {}, + "remote_url_pattern": {}, + "source_discover_config": {}, + "analyse_config": {}, + }, + "additionalProperties": False, + }, + }, + }, + ) + """The configuration for the source tracing projects.""" + + debug_measurement: bool = field( + default=False, metadata={"rebuild": "html", "types": (bool,)} + ) + """If True, log runtime information for various functions.""" + debug_filters: bool = field( + default=False, metadata={"rebuild": "html", "types": (bool,)} + ) + """If True, log filter processing runtime information.""" + + +def check_schema(config: CodeLinksConfig) -> list[str]: + """Check only first layer's of schema, so that the nested dict is not validated here.""" + errors = [] + for _field_name in CodeLinksConfig.field_names(): + schema = CodeLinksConfig.get_schema(_field_name) + if not schema: + continue + value = getattr(config, _field_name) + if isinstance(value, Path): # adapt to json schema restriction + value = str(value) + try: + validate(instance=value, schema=schema) + except ValidationError as e: + errors.append( + f"Schema validation error in filed '{_field_name}': {e.message}" + ) + return errors + + +def check_project_configuration(config: CodeLinksConfig) -> list[str]: + """Check nested project configurations""" + errors = [] + + for project_name, project_config in config.projects.items(): + project_errors: list[str] = [] + + # validate source_discover config + src_discover_config: SourceDiscoverConfig | None = project_config.get( + "source_discover_config" + ) + src_discover_errors = [] + if src_discover_config: + src_discover_errors.extend(src_discover_config.check_schema()) + + # validate analyse config + analyse_config: SourceAnalyseConfig | None = project_config.get( + "analyse_config" + ) + analyse_errors = [] + if analyse_config: + analyse_errors = analyse_config.check_fields_configuration() + + # validate src-trace config + if config.set_remote_url and "remote_url_pattern" not in project_config: + project_errors.append( + "remote_url_pattern must be given, as set_remote_url is enabled" + ) + + if "remote_url_pattern" in project_config and not isinstance( + project_config["remote_url_pattern"], str + ): + project_errors.append("remote_url_pattern must be a string") + + if analyse_errors or src_discover_errors or project_errors: + errors.append(f"Project '{project_name}' has the following errors:") + errors.extend(analyse_errors) + errors.extend(src_discover_errors) + errors.extend(project_errors) + + return errors + + +def check_configuration(config: CodeLinksConfig) -> list[str]: + errors = [] + errors.extend(check_schema(config)) + errors.extend(check_project_configuration(config)) + return errors + + +def convert_src_discovery_config( + config_dict: SourceDiscoverSectionConfigType | None, +) -> SourceDiscoverConfig: + if config_dict: + src_discover_dict = { + key: (Path(value) if key == "src_dir" and isinstance(value, str) else value) + for key, value in config_dict.items() + } + src_discover_config = SourceDiscoverConfig(**src_discover_dict) # type: ignore[arg-type] # mypy is confused by dynamic assignment + else: + src_discover_config = SourceDiscoverConfig() + + return src_discover_config + + +def convert_analyse_config( + config_dict: AnalyseSectionConfigType | None, + src_discover: SourceDiscover | None = None, +) -> SourceAnalyseConfig: + analyse_config_dict: SourceAnalyseConfigType = {} + if config_dict: + for k, v in config_dict.items(): + if k not in {"online_comment_style", "need_id_refs", "marked_rst"}: + analyse_config_dict[k] = ( # type: ignore[literal-required] # dynamical assignment + Path(v) if k == "src_dic" and isinstance(v, str) else v + ) + + # Get oneline_comment_style configuration + oneline_comment_style_dict: OneLineCommentStyleType | None = config_dict.get( + "oneline_comment_style" + ) + oneline_comment_style: OneLineCommentStyle = ( + convert_oneline_comment_style_config(oneline_comment_style_dict) + ) + + # Get need_id_refs configuration + need_id_refs_config_dict: NeedIdRefsConfigType | None = config_dict.get( + "need_id_refs" + ) + need_id_refs_config = convert_need_id_refs_config(need_id_refs_config_dict) + + # Get marked_rst configuration + marked_rst_config_dict: MarkedRstConfigType | None = config_dict.get( + "marked_rst" + ) + marked_rst_config = convert_marked_rst_config(marked_rst_config_dict) + + analyse_config_dict["need_id_refs_config"] = need_id_refs_config + analyse_config_dict["marked_rst_config"] = marked_rst_config + analyse_config_dict["oneline_comment_style"] = oneline_comment_style + + if src_discover: + analyse_config_dict["src_files"] = src_discover.source_paths + analyse_config_dict["src_dir"] = src_discover.src_discover_config.src_dir + + return SourceAnalyseConfig(**analyse_config_dict) + + +def convert_oneline_comment_style_config( + config_dict: OneLineCommentStyleType | None, +) -> OneLineCommentStyle: + if config_dict is None: + oneline_comment_style = OneLineCommentStyle() + else: + try: + oneline_comment_style = OneLineCommentStyle(**config_dict) + except TypeError as e: + raise TypeError(f"Invalid oneline comment style configuration: {e}") from e + return oneline_comment_style + + +def convert_need_id_refs_config( + config_dict: NeedIdRefsConfigType | None, +) -> NeedIdRefsConfig: + if not config_dict: + need_id_refs_config = NeedIdRefsConfig() + else: + try: + need_id_refs_config = NeedIdRefsConfig(**config_dict) + except TypeError as e: + raise TypeError(f"Invalid oneline comment style configuration: {e}") from e + return need_id_refs_config + + +def convert_marked_rst_config( + config_dict: MarkedRstConfigType | None, +) -> MarkedRstConfig: + if not config_dict: + marked_rst_config = MarkedRstConfig() + else: + try: + marked_rst_config = MarkedRstConfig(**config_dict) + except TypeError as e: + raise TypeError(f"Invalid oneline comment style configuration: {e}") from e + return marked_rst_config + + +def generate_project_configs( + project_configs: dict[str, CodeLinksProjectConfigType], +) -> None: + """Generate configs of source discover and analyse from their classes dynamically.""" + for project_config in project_configs.values(): + # overwrite the config into different types on purpose + # covert dicts to their own classes + src_discover_section: SourceDiscoverSectionConfigType | None = cast( + SourceDiscoverSectionConfigType, + project_config.get("source_discover"), + ) + source_discover_config = convert_src_discovery_config(src_discover_section) + project_config["source_discover_config"] = source_discover_config + + analyse_section_config: AnalyseSectionConfigType | None = cast( + AnalyseSectionConfigType, project_config.get("analyse") + ) + analyse_config = convert_analyse_config(analyse_section_config) + analyse_config.get_oneline_needs = True # force to get oneline_need + project_config["analyse_config"] = analyse_config diff --git a/src/sphinx_codelinks/source_discover/config.py b/src/sphinx_codelinks/source_discover/config.py index d37d854..687f77a 100644 --- a/src/sphinx_codelinks/source_discover/config.py +++ b/src/sphinx_codelinks/source_discover/config.py @@ -1,18 +1,36 @@ from dataclasses import MISSING, dataclass, field, fields +from enum import Enum from pathlib import Path from typing import Any, Required, TypedDict, cast from jsonschema import ValidationError, validate -from sphinx_codelinks.analyse.config import SUPPORTED_COMMENT_TYPES +COMMENT_FILETYPE = {"cpp": ["c", "cpp", "h", "hpp"], "python": ["py"]} + + +class CommentType(str, Enum): + python = "python" + cpp = "cpp" + + +class SourceDiscoverSectionConfigType(TypedDict, total=False): + """Define typing for loading configuration from TOML files""" + + src_dir: Required[str] + exclude: list[str] + include: list[str] + gitignore: bool + comment_type: CommentType class SourceDiscoverConfigType(TypedDict, total=False): + """Define typing for its API configuration""" + src_dir: Required[Path] exclude: list[str] include: list[str] gitignore: bool - file_types: list[str] + comment_type: CommentType @dataclass @@ -41,12 +59,12 @@ def field_names(cls) -> set[str]: gitignore: bool = field(default=True, metadata={"schema": {"type": "boolean"}}) """Whether to respect .gitignore to exclude files.""" - file_types: list[str] = field( - default_factory=lambda: list(SUPPORTED_COMMENT_TYPES), + comment_type: str = field( + default="cpp", metadata={ "schema": { - "type": "array", - "items": {"type": "string", "enum": sorted(SUPPORTED_COMMENT_TYPES)}, + "type": "string", + "enum": sorted(COMMENT_FILETYPE), } }, ) diff --git a/src/sphinx_codelinks/source_discover/source_discover.py b/src/sphinx_codelinks/source_discover/source_discover.py index d9e336c..84c5563 100644 --- a/src/sphinx_codelinks/source_discover/source_discover.py +++ b/src/sphinx_codelinks/source_discover/source_discover.py @@ -7,7 +7,10 @@ parse_gitignore, ) -from sphinx_codelinks.source_discover.config import SourceDiscoverConfig +from sphinx_codelinks.source_discover.config import ( + COMMENT_FILETYPE, + SourceDiscoverConfig, +) class SourceDiscover: @@ -22,16 +25,9 @@ def __init__(self, src_discover_config: SourceDiscoverConfig): else None ) # normalize the file types to lower case with leading dot - self.file_types = ( - { - file_type.lower() - if file_type.startswith(".") - else f".{file_type}".lower() - for file_type in src_discover_config.file_types - } - if src_discover_config.file_types - else None - ) + self.file_types = { + f".{ext}" for ext in COMMENT_FILETYPE[src_discover_config.comment_type] + } self.source_paths = self._discover() diff --git a/src/sphinx_codelinks/sphinx_extension/config.py b/src/sphinx_codelinks/sphinx_extension/config.py deleted file mode 100644 index 9310981..0000000 --- a/src/sphinx_codelinks/sphinx_extension/config.py +++ /dev/null @@ -1,329 +0,0 @@ -from dataclasses import MISSING, dataclass, field, fields -from pathlib import Path -from typing import Any, TypedDict, cast - -from jsonschema import ValidationError, validate -from sphinx.application import Sphinx -from sphinx.config import Config as _SphinxConfig - -from sphinx_codelinks.analyse.config import ( - SUPPORTED_COMMENT_TYPES, - CommentType, - MarkedRstConfigType, - NeedIdRefsConfigType, - OneLineCommentStyle, - OneLineCommentStyleType, -) -from sphinx_codelinks.source_discover.config import ( - SourceDiscoverConfig, - SourceDiscoverConfigType, -) - -SRC_TRACE_CACHE: str = "src_trace_cache" - - -class SourceTracingLineHref: - """Global class for the mapping between source file line numbers and Sphinx documentation links.""" - - def __init__(self) -> None: - self.mappings: dict[str, dict[int, str]] = {} - - -file_lineno_href = SourceTracingLineHref() - - -class SrcTraceProjectConfigFileType(TypedDict, total=False): - # only support C/C++ for now - comment_type: CommentType - src_dir: str - remote_url_pattern: str - exclude: list[str] - include: list[str] - gitignore: bool - oneline_comment_style: OneLineCommentStyleType - need_id_refs: NeedIdRefsConfigType - marked_rst: MarkedRstConfigType - - -class SrcTraceProjectConfigType(TypedDict): - # only support C/C++ for now - comment_type: CommentType - src_dir: Path - remote_url_pattern: str - exclude: list[str] - include: list[str] - gitignore: bool - oneline_comment_style: OneLineCommentStyle - - -class SrcTraceConfigType(TypedDict): - config_from_toml: str | None - set_local_url: bool - local_url_field: str - set_remote_url: bool - remote_url_field: str - projects: dict[str, SrcTraceProjectConfigType] - debug_measurement: bool - debug_filters: bool - - -@dataclass -class SrcTraceSphinxConfig: - def __init__(self, config: _SphinxConfig) -> None: - super().__setattr__("_config", config) - - def __getattribute__(self, name: str) -> Any: # type: ignore[explicit-any] - if name.startswith("__") or name == "_config": - return super().__getattribute__(name) - return getattr(super().__getattribute__("_config"), f"src_trace_{name}") - - def __setattr__(self, name: str, value: Any) -> None: # type: ignore[explicit-any] - if name == "_config" and "src_trace_projects" in value: - src_trace_projects: dict[str, SrcTraceProjectConfigType] = value[ - "src_trace_projects" - ] - for _config in src_trace_projects.values(): - # overwrite the config into different types on purpose - # covert dict to OneLineCommentStyle class - oneline_comment_style: OneLineCommentStyleType | None = cast( - OneLineCommentStyleType, _config.get("oneline_comment_style") - ) - if not oneline_comment_style: - raise Exception("OneLineCommentStyle is not given") - - _config["oneline_comment_style"] = OneLineCommentStyle( - **oneline_comment_style - ) - if name.startswith("__") or name == "_config": - return super().__setattr__(name, value) - - return setattr(super().__getattribute__("_config"), f"src_trace_{name}", value) - - @classmethod - def add_config_values(cls, app: Sphinx) -> None: - """Add all config values to Sphinx application""" - for item in fields(cls): - if item.default_factory is not MISSING: - default = item.default_factory() - elif item.default is not MISSING: - default = item.default - else: - raise Exception(f"Field {item.name} has no default value or factory") - - name = item.name - app.add_config_value( - f"src_trace_{name}", - default, - item.metadata["rebuild"], - types=item.metadata["types"], - ) - - @classmethod - def field_names(cls) -> set[str]: - return {item.name for item in fields(cls)} - - @classmethod - def get_schema(cls, name: str) -> dict[str, Any] | None: # type: ignore[explicit-any] - """Get the schema for a config item.""" - _field = next(field for field in fields(cls) if field.name is name) - if _field.metadata is not MISSING and "schema" in _field.metadata: - return _field.metadata["schema"] # type: ignore[no-any-return] - return None - - config_from_toml: str | None = field( - default=None, - metadata={ - "rebuild": "env", - "types": (str, type(None)), - "schema": { - "type": ["string", "null"], - "examples": ["config.toml", None], - }, - }, - ) - """Path to a TOML file to load configuration from.""" - - set_local_url: bool = field( - default=False, - metadata={ - "rebuild": "env", - "types": (bool,), - "schema": { - "type": "boolean", - }, - }, - ) - """Set the file URL in the extracted need.""" - - local_url_field: str = field( - default="local-url", - metadata={ - "rebuild": "env", - "types": (str,), - "schema": { - "type": "string", - }, - }, - ) - """The field name for the file URL in the extracted need.""" - - set_remote_url: bool = field( - default=False, - metadata={ - "rebuild": "env", - "types": (bool,), - "schema": { - "type": "boolean", - }, - }, - ) - remote_url_field: str = field( - default="remote-url", - metadata={ - "rebuild": "env", - "types": (str,), - "schema": { - "type": "string", - }, - }, - ) - """The field name for the remote URL in the extracted need.""" - - projects: dict[str, SrcTraceProjectConfigType] = field( - default_factory=dict, - metadata={ - "rebuild": "env", - "types": (), - "schema": { - "type": "object", - "additionalProperties": { - "type": "object", - "properties": { - "comment_type": {}, - "src_dir": {}, - "remote_url_pattern": {}, - "exclude": {}, - "include": {}, - "gitignore": {}, - "oneline_comment_style": {}, - }, - "additionalProperties": False, - }, - }, - }, - ) - """The configuration for the source tracing projects.""" - - debug_measurement: bool = field( - default=False, metadata={"rebuild": "html", "types": (bool,)} - ) - """If True, log runtime information for various functions.""" - debug_filters: bool = field( - default=False, metadata={"rebuild": "html", "types": (bool,)} - ) - """If True, log filter processing runtime information.""" - - -def check_schema(config: SrcTraceSphinxConfig) -> list[str]: - errors = [] - for _field_name in SrcTraceSphinxConfig.field_names(): - schema = SrcTraceSphinxConfig.get_schema(_field_name) - value = getattr(config, _field_name) - if not schema: - continue - try: - validate(instance=value, schema=schema) - except ValidationError as e: - errors.append( - f"Schema validation error in filed '{_field_name}': {e.message}" - ) - return errors - - -def check_project_configuration(config: SrcTraceSphinxConfig) -> list[str]: - errors = [] - - for project_name, project_config in config.projects.items(): - project_errors: list[str] = [] - oneline_errors = validate_oneline_comment_style(project_config) - src_discover_dict = build_src_discover_dict(project_config) - src_discover_errors = [] - if src_discover_dict is not None: - src_discover_config = SourceDiscoverConfig(**src_discover_dict) - src_discover_errors.extend(src_discover_config.check_schema()) - - if config.set_remote_url and "remote_url_pattern" not in project_config: - project_errors.append( - "remote_url_pattern must be given, as set_remote_url is enabled" - ) - - if "remote_url_pattern" in project_config and not isinstance( - project_config["remote_url_pattern"], str - ): - project_errors.append("remote_url_pattern must be a string") - - if oneline_errors or src_discover_errors or project_errors: - errors.append(f"Project '{project_name}' has the following errors:") - errors.extend(oneline_errors) - errors.extend(src_discover_errors) - errors.extend(project_errors) - - return errors - - -def check_configuration(config: SrcTraceSphinxConfig) -> list[str]: - errors = [] - errors.extend(check_schema(config)) - errors.extend(check_project_configuration(config)) - return errors - - -def validate_oneline_comment_style( - project_config: SrcTraceProjectConfigType, -) -> list[str]: - if "oneline_comment_style" in project_config: - style = project_config["oneline_comment_style"] - if isinstance(style, OneLineCommentStyle): - return style.check_fields_configuration() - return [] - - -def build_src_discover_dict( - project_config: SrcTraceProjectConfigType, -) -> SourceDiscoverConfigType | None: - src_discover_dict = cast(SourceDiscoverConfigType, {}) - # adapt the configs between source tracing and source discovery - if "comment_type" in project_config: - # comment type error will be taken care by SourceDiscovery class later - src_discover_dict["file_types"] = ( - list(SUPPORTED_COMMENT_TYPES) - if project_config["comment_type"] in SUPPORTED_COMMENT_TYPES - else [project_config["comment_type"]] - ) - for key in ("exclude", "include", "gitignore", "src_dir"): - if key in project_config: - src_discover_dict[key] = project_config[key] - - return src_discover_dict - - -def adpat_src_discover_config(project_config: SrcTraceProjectConfigType) -> None: - src_discover_dict = build_src_discover_dict(project_config) - if src_discover_dict: - src_discover_config = SourceDiscoverConfig(**src_discover_dict) - else: - src_discover_config = SourceDiscoverConfig() - - for _field in fields(src_discover_config): - key = "comment_type" if _field.name == "file_types" else _field.name - - if key == "comment_type": - file_types = getattr(src_discover_config, _field.name) - if set(file_types) == SUPPORTED_COMMENT_TYPES: - comment_type = CommentType.cpp - else: - comment_type = file_types[0] - project_config[key] = comment_type # type: ignore[literal-required] # dynamically assign - continue - - project_config[key] = getattr(src_discover_config, _field.name) # type: ignore[literal-required] # dynamically assign diff --git a/src/sphinx_codelinks/sphinx_extension/directives/src_trace.py b/src/sphinx_codelinks/sphinx_extension/directives/src_trace.py index 65ed091..43a7364 100644 --- a/src/sphinx_codelinks/sphinx_extension/directives/src_trace.py +++ b/src/sphinx_codelinks/sphinx_extension/directives/src_trace.py @@ -12,15 +12,14 @@ from sphinx_needs.utils import add_doc # type: ignore[import-untyped] from sphinx_codelinks.analyse.analyse import SourceAnalyse -from sphinx_codelinks.analyse.config import COMMENT_FILETYPE, SourceAnalyseConfig from sphinx_codelinks.analyse.models import OneLineNeed -from sphinx_codelinks.source_discover.config import SourceDiscoverConfig -from sphinx_codelinks.source_discover.source_discover import SourceDiscover -from sphinx_codelinks.sphinx_extension.config import ( - SrcTraceProjectConfigType, - SrcTraceSphinxConfig, +from sphinx_codelinks.config import ( + CodeLinksConfig, + CodeLinksProjectConfigType, file_lineno_href, ) +from sphinx_codelinks.source_discover.config import SourceDiscoverConfig +from sphinx_codelinks.source_discover.source_discover import SourceDiscover from sphinx_codelinks.sphinx_extension.debug import measure_time sphinx_version = sphinx.__version__ @@ -87,40 +86,30 @@ def run(self) -> list[nodes.Node]: project = self.options["project"] title = self.arguments[0] # get source tracing config - src_trace_sphinx_config = SrcTraceSphinxConfig(self.env.config) + src_trace_sphinx_config = CodeLinksConfig.from_sphinx(self.env.config) # load config - src_trace_conf: SrcTraceProjectConfigType = src_trace_sphinx_config.projects[ + src_trace_conf: CodeLinksProjectConfigType = src_trace_sphinx_config.projects[ project ] - comment_type = src_trace_conf["comment_type"] - oneline_comment_style = src_trace_conf["oneline_comment_style"] - - src_dir = self.locate_src_dir(src_trace_sphinx_config, src_trace_conf) + src_discover_config = src_trace_conf["source_discover_config"] + src_dir = self.locate_src_dir(src_trace_sphinx_config, src_discover_config) out_dir = Path(self.env.app.outdir) # the directory where the source files are copied to target_dir = out_dir / src_dir.name extra_options = {"project": project} - source_files = self.get_src_files(self.options, src_dir, src_trace_conf) + source_files = self.get_src_files(self.options, src_dir, src_discover_config) # add source files into the dependency # https://www.sphinx-doc.org/en/master/extdev/envapi.html#sphinx.environment.BuildEnvironment.note_dependency for source_file in source_files: self.env.note_dependency(str(source_file.resolve())) - analyse_config = SourceAnalyseConfig( - source_files, - src_dir, - out_dir, - comment_type, - get_need_id_refs=False, - get_oneline_needs=True, - get_rst=False, - oneline_comment_style=oneline_comment_style, - ) - + analyse_config = src_trace_conf["analyse_config"] + analyse_config.src_dir = src_dir + analyse_config.src_files = source_files src_analyse = SourceAnalyse(analyse_config) src_analyse.run() @@ -204,8 +193,9 @@ def get_src_files( self, extra_options: dict[str, str], src_dir: Path, - src_trace_conf: SrcTraceProjectConfigType, + src_discover_config: SourceDiscoverConfig, ) -> list[Path]: + """Leverage SourceDiscover to find sources files from the given directory.""" source_files = [] if "file" in self.options: file: str = self.options["file"] @@ -220,13 +210,13 @@ def get_src_files( else: extra_options["directory"] = directory dir_path = src_dir / directory - file_types = COMMENT_FILETYPE[src_trace_conf["comment_type"]] + # create a new config for the specified directory src_discover = SourceDiscoverConfig( dir_path, - gitignore=src_trace_conf["gitignore"], - include=src_trace_conf["include"], - exclude=src_trace_conf["exclude"], - file_types=file_types, + gitignore=src_discover_config.gitignore, + include=src_discover_config.include, + exclude=src_discover_config.exclude, + comment_type=src_discover_config.comment_type, ) source_discover = SourceDiscover(src_discover) source_files.extend(source_discover.source_paths) @@ -235,8 +225,8 @@ def get_src_files( def locate_src_dir( self, - src_trace_sphinx_config: SrcTraceSphinxConfig, - src_trace_conf: SrcTraceProjectConfigType, + src_trace_sphinx_config: CodeLinksConfig, + src_discover_config: SourceDiscoverConfig, ) -> Path: """Locate the source directory based on the configuration.""" # src dir in src_trace_conf is relative to conf_dir by default @@ -246,7 +236,7 @@ def locate_src_dir( src_trace_toml_path = Path(src_trace_sphinx_config.config_from_toml) conf_dir = conf_dir / src_trace_toml_path.parent - src_dir = (conf_dir / src_trace_conf["src_dir"]).resolve() + src_dir = (conf_dir / src_discover_config.src_dir).resolve() return src_dir def render_needs( diff --git a/src/sphinx_codelinks/sphinx_extension/source_tracing.py b/src/sphinx_codelinks/sphinx_extension/source_tracing.py index e687cb3..5a8f956 100644 --- a/src/sphinx_codelinks/sphinx_extension/source_tracing.py +++ b/src/sphinx_codelinks/sphinx_extension/source_tracing.py @@ -15,18 +15,17 @@ add_need_type, ) -from sphinx_codelinks.analyse.analyse import SourceAnalyse -from sphinx_codelinks.analyse.config import OneLineCommentStyle, OneLineCommentStyleType -from sphinx_codelinks.sphinx_extension import debug -from sphinx_codelinks.sphinx_extension.config import ( +from sphinx_codelinks.analyse.projects import AnalyseProjects +from sphinx_codelinks.config import ( SRC_TRACE_CACHE, - SrcTraceConfigType, - SrcTraceProjectConfigType, - SrcTraceSphinxConfig, - adpat_src_discover_config, + CodeLinksConfig, + CodeLinksConfigType, + CodeLinksProjectConfigType, check_configuration, file_lineno_href, + generate_project_configs, ) +from sphinx_codelinks.sphinx_extension import debug from sphinx_codelinks.sphinx_extension.directives.src_trace import ( SourceTracing, SourceTracingDirective, @@ -39,7 +38,7 @@ def setup(app: Sphinx) -> dict[str, Any]: # type: ignore[explicit-any] app.add_node(SourceTracing) app.add_directive("src-trace", SourceTracingDirective) - SrcTraceSphinxConfig.add_config_values(app) + CodeLinksConfig.add_config_values(app) app.connect("config-inited", load_config_from_toml, priority=10) app.connect( @@ -111,7 +110,7 @@ def generate_code_page( def load_config_from_toml(app: Sphinx, config: _SphinxConfig) -> None: """Load the configuration from a TOML file, if defined in conf.py.""" - src_trc_sphinx_config = SrcTraceSphinxConfig(config) + src_trc_sphinx_config = CodeLinksConfig.from_sphinx(config) if src_trc_sphinx_config.config_from_toml is None: return @@ -127,7 +126,7 @@ def load_config_from_toml(app: Sphinx, config: _SphinxConfig) -> None: try: with toml_file.open("rb") as f: toml_data = tomllib.load(f) - toml_data = toml_data["src_trace"] + toml_data = toml_data["codelinks"] if not isinstance(toml_data, dict): raise Exception(f"data must be a dict in {toml_file}") @@ -138,43 +137,27 @@ def load_config_from_toml(app: Sphinx, config: _SphinxConfig) -> None: return set_config_to_sphinx( - src_trace_config=cast(SrcTraceConfigType, toml_data), config=config + src_trace_config=cast(CodeLinksConfigType, toml_data), config=config ) def set_config_to_sphinx( - src_trace_config: SrcTraceConfigType, config: _SphinxConfig + src_trace_config: CodeLinksConfigType, config: _SphinxConfig ) -> None: - allowed_keys = SrcTraceSphinxConfig.field_names() + allowed_keys = CodeLinksConfig.field_names() for key, value in src_trace_config.items(): if key not in allowed_keys: continue if key == "projects": - for project_config in cast( - dict[str, SrcTraceProjectConfigType], value - ).values(): - # address SourceDiscovery related config - adpat_src_discover_config(project_config) - - # address OneLoneCommenyStyle config and its default - oneline_comment_style: OneLineCommentStyleType | None = cast( - OneLineCommentStyleType, project_config.get("oneline_comment_style") - ) - if oneline_comment_style: - project_config["oneline_comment_style"] = OneLineCommentStyle( - **cast( - OneLineCommentStyleType, - project_config["oneline_comment_style"], - ) - ) - else: - project_config["oneline_comment_style"] = OneLineCommentStyle() - + src_trace_projects: dict[str, CodeLinksProjectConfigType] = cast( + dict[str, CodeLinksProjectConfigType], value + ) + generate_project_configs(src_trace_projects) config[f"src_trace_{key}"] = value def update_sn_extra_options(app: Sphinx, config: _SphinxConfig) -> None: - src_trace_sphinx_config = SrcTraceSphinxConfig(config) + src_trace_sphinx_config = CodeLinksConfig.from_sphinx(config) add_extra_option(app, "project") add_extra_option(app, "file") add_extra_option(app, "directory") @@ -192,7 +175,7 @@ def prepare_env(app: Sphinx, env: BuildEnvironment, _docnames: list[str]) -> Non """ Prepares the sphinx environment to store stc-trace internal data. """ - src_trace_sphinx_config = SrcTraceSphinxConfig(app.config) + src_trace_sphinx_config = CodeLinksConfig.from_sphinx(app.config) # Set time measurement flag if src_trace_sphinx_config.debug_measurement: @@ -205,7 +188,7 @@ def prepare_env(app: Sphinx, env: BuildEnvironment, _docnames: list[str]) -> Non def check_sphinx_configuration(app: Sphinx, _config: _SphinxConfig) -> None: - config = SrcTraceSphinxConfig(app.config) + config = CodeLinksConfig.from_sphinx(app.config) errors = check_configuration(config) if errors: raise Exception("\n".join(errors)) @@ -215,7 +198,7 @@ def emit_warnings( app: Sphinx, _env: BuildEnvironment, ) -> None: - warnings = SourceAnalyse.load_warnings(Path(app.outdir) / SRC_TRACE_CACHE) + warnings = AnalyseProjects.load_warnings(Path(app.outdir) / SRC_TRACE_CACHE) if not warnings: return for warning in warnings: diff --git a/tests/conftest.py b/tests/conftest.py index 819d1c0..3930bcb 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -5,17 +5,13 @@ import pytest from syrupy.extensions.single_file import SingleFileSnapshotExtension, WriteMode -from sphinx_codelinks.analyse.config import OneLineCommentStyle +from sphinx_codelinks.config import OneLineCommentStyle pytest_plugins = "sphinx.testing.fixtures" TEST_DIR = Path(__file__).parent DATA_DIR = TEST_DIR / "data" SRC_TRACE_TOML = TEST_DIR / "data" / "sphinx" / "src_trace.toml" -BASIC_analyse_TOML = TEST_DIR / "data" / "oneline_comment_basic" / "analyse_config.toml" -DEFAULT_analyse_TOML = ( - TEST_DIR / "data" / "oneline_comment_default" / "analyse_config.toml" -) RECURSIVE_DIR_analyse_TOML = TEST_DIR / "doc_test" / "recursive_dirs" / "src_trace.toml" ONELINE_COMMENT_STYLE = OneLineCommentStyle( start_sequence="[[", diff --git a/tests/data/analyse/default_config.toml b/tests/data/analyse/default_config.toml deleted file mode 100644 index 24117a8..0000000 --- a/tests/data/analyse/default_config.toml +++ /dev/null @@ -1,3 +0,0 @@ -[source_discover] -src_dir = "../../data" -gitignore = false diff --git a/tests/data/configs/full_config.toml b/tests/data/configs/full_config.toml new file mode 100644 index 0000000..4b5adb5 --- /dev/null +++ b/tests/data/configs/full_config.toml @@ -0,0 +1,43 @@ +[codelinks] +# Configuration for source tracing +set_local_url = true # Set to true to enable local code html and URL generation +local_url_field = "local-url" # Need's field name for local URL +set_remote_url = true # Set to true to enable remote url to be generated +remote_url_field = "remote-url" # Need's field name for remote URL +outdir = "codelinks_output" # Output directory for generated files + +# Configuration for source tracing project "dcdc" +[codelinks.projects.dcdc] +remote_url_pattern = "https://github.com/useblocks/sphinx-codelinks/blob/{commit}/{path}#L{line}" # URL pattern for remote source code + +[codelinks.projects.dcdc.source_discover] +src_dir = "../dcdc" # Relative path from this TOML config to the source directory +comment_type = "cpp" +exclude = [] +include = ["**/*.c", "**/*.h"] +gitignore = false + +[codelinks.projects.dcdc.analyse] +get_need_id_refs = false +get_oneline_needs = true +get_rst = false + +[codelinks.projects.dcdc.analyse.oneline_comment_style] +# Configuration for oneline comment style +start_sequence = "[[" # Start sequence for oneline comments +end_sequence = "]]" # End sequence for the online comments; default is newline character +field_split_char = "," # Character to split fields in the comment +# Fields that are defined in the oneline comment style +needs_fields = [ + { "name" = "id", "type" = "str" }, + { "name" = "title", "type" = "str" }, + { "name" = "type", "type" = "str", "default" = "impl" }, + { "name" = "links", "type" = "list[str]", "default" = [ + ] }, +] + +[codelinks.projects.src] +remote_url_pattern = "https://github.com/useblocks/sphinx-codelinks/blob/{commit}/{path}#L{line}" + +[codelinks.projects.src.source_discover] +src_dir = "../doc_test/minimum_config" diff --git a/tests/data/configs/minimum_config.toml b/tests/data/configs/minimum_config.toml new file mode 100644 index 0000000..33484f8 --- /dev/null +++ b/tests/data/configs/minimum_config.toml @@ -0,0 +1,8 @@ +[codelinks.projects.minimum_config.source_discover] +src_dir = "../../data" +gitignore = false + +[codelinks.projects.minimum_config.analyse] +get_need_id_refs = true +get_oneline_needs = false +get_rst = true diff --git a/tests/data/oneline_comment_basic/analyse_config.toml b/tests/data/oneline_comment_basic/analyse_config.toml index 70c64fc..6ceefba 100644 --- a/tests/data/oneline_comment_basic/analyse_config.toml +++ b/tests/data/oneline_comment_basic/analyse_config.toml @@ -1,11 +1,11 @@ -[source_discover] +[codelinks.projects.basic.source_discover] src_dir = "./" -comment_type = "c" +comment_type = "cpp" exclude = [] include = ["**/*.c", "**/*.h"] gitignore = false -[oneline_comment_style] +[codelinks.projects.basic.analyse.oneline_comment_style] start_sequence = "[[" end_sequence = "]]" # default is newline character field_split_char = "," diff --git a/tests/data/oneline_comment_default/analyse_config.toml b/tests/data/oneline_comment_default/analyse_config.toml index c128991..fb536d6 100644 --- a/tests/data/oneline_comment_default/analyse_config.toml +++ b/tests/data/oneline_comment_default/analyse_config.toml @@ -1,6 +1,6 @@ -[source_discover] +[codelinks.projects.default.source_discover] src_dir = "./" -comment_type = "c" +comment_type = "cpp" exclude = [] include = ["**/*.c", "**/*.h"] gitignore = false diff --git a/tests/data/sphinx/src_trace.toml b/tests/data/sphinx/src_trace.toml index cba8aa2..c19e368 100644 --- a/tests/data/sphinx/src_trace.toml +++ b/tests/data/sphinx/src_trace.toml @@ -1,19 +1,23 @@ -[src_trace] +[codelinks] set_local_url = true local_url_field = "local-url" set_remote_url = true remote_url_field = "remote-url" debug_measurement = true -[src_trace.projects.dcdc] +[codelinks.projects.dcdc] + +remote_url_pattern = "https://github.com/useblocks/sphinx-codelinks/blob/{commit}/{path}#L{line}" + +[codelinks.projects.dcdc.source_discover] comment_type = "cpp" src_dir = "../dcdc" -remote_url_pattern = "https://github.com/useblocks/sphinx-codelinks/blob/{commit}/{path}#L{line}" exclude = ["dcdc/src/ubt/ubt.cpp"] include = ["**/*.cpp", "**/*.hpp"] gitignore = true -[src_trace.projects.dcdc.oneline_comment_style] + +[codelinks.projects.dcdc.analyse.oneline_comment_style] start_sequence = "[[" end_sequence = "]]" # default is newline character field_split_char = "," diff --git a/tests/doc_test/minimum_config/src_trace.toml b/tests/doc_test/minimum_config/src_trace.toml index da414a4..0561780 100644 --- a/tests/doc_test/minimum_config/src_trace.toml +++ b/tests/doc_test/minimum_config/src_trace.toml @@ -1,2 +1,2 @@ -[src_trace.projects.src] +[codelinks.projects.src] remote_url_pattern = "https://github.com/useblocks/sphinx-codelinks/blob/{commit}/{path}#L{line}" diff --git a/tests/doc_test/recursive_dirs/src_trace.toml b/tests/doc_test/recursive_dirs/src_trace.toml index fc921f4..b3e38a3 100644 --- a/tests/doc_test/recursive_dirs/src_trace.toml +++ b/tests/doc_test/recursive_dirs/src_trace.toml @@ -1,14 +1,16 @@ -[src_trace] +[codelinks] set_local_url = true local_url_field = "local-url" set_remote_url = true remote_url_field = "remote-url" debug_measurement = true -[src_trace.projects.dummy_src] +[codelinks.projects.dummy_src] +remote_url_pattern = "https://github.com/useblocks/sphinx-codelinks/blob/{commit}/{path}#L{line}" + +[codelinks.projects.dummy_src.source_discover] comment_type = "cpp" src_dir = "./dummy_src_lv1" -remote_url_pattern = "https://github.com/useblocks/sphinx-codelinks/blob/{commit}/{path}#L{line}" exclude = ["dcdc/src/ubt/ubt.cpp"] include = ["**/*.cpp", "**/*.hpp"] gitignore = true diff --git a/tests/test_analyse.py b/tests/test_analyse.py index 6512735..d9897cc 100644 --- a/tests/test_analyse.py +++ b/tests/test_analyse.py @@ -4,7 +4,7 @@ import pytest from sphinx_codelinks.analyse.analyse import SourceAnalyse -from sphinx_codelinks.analyse.config import SourceAnalyseConfig +from sphinx_codelinks.config import SourceAnalyseConfig from tests.conftest import ( ONELINE_COMMENT_STYLE, ONELINE_COMMENT_STYLE_DEFAULT, @@ -31,16 +31,16 @@ def test_analyse(src_dir, src_paths, tmp_path, snapshot_marks): src_analyse_config = SourceAnalyseConfig( src_files=src_paths, src_dir=src_dir, - outdir=tmp_path, get_need_id_refs=True, get_oneline_needs=True, get_rst=True, ) - analyser = SourceAnalyse(src_analyse_config) - analyser.git_remote_url = None - analyser.git_commit_rev = None - analyser.run() + analyse = SourceAnalyse(src_analyse_config) + analyse.git_remote_url = None + analyse.git_commit_rev = None + analyse.run() + analyse.dump_marked_content(tmp_path) dumped_content = tmp_path / "marked_content.json" with dumped_content.open("r") as f: @@ -111,7 +111,6 @@ def test_analyse_oneline_needs( src_analyse_config = SourceAnalyseConfig( src_files=src_paths, src_dir=src_dir, - outdir=tmp_path, get_need_id_refs=False, get_oneline_needs=True, get_rst=False, @@ -122,14 +121,8 @@ def test_analyse_oneline_needs( assert len(src_analyse.src_files) == result["num_src_files"] assert len(src_analyse.oneline_warnings) == result["num_oneline_warnings"] - assert src_analyse.warnings_path.exists() - - loaded_warnings = SourceAnalyse.load_warnings(tmp_path) cnt_comments = 0 for src_file in src_analyse.src_files: cnt_comments += len(src_file.src_comments) assert cnt_comments == result["num_comments"] - - # use cache - assert SourceAnalyse.load_warnings(tmp_path) == loaded_warnings diff --git a/tests/test_analyse_config.py b/tests/test_analyse_config.py index dd14542..7c81d5d 100644 --- a/tests/test_analyse_config.py +++ b/tests/test_analyse_config.py @@ -1,12 +1,12 @@ import pytest -from sphinx_codelinks.analyse.config import OneLineCommentStyle, SourceAnalyseConfig +from sphinx_codelinks.config import OneLineCommentStyle, SourceAnalyseConfig from .conftest import TEST_DIR @pytest.mark.parametrize( - ("vdocs_config", "result"), + ("analyse_config", "result"), [ ( SourceAnalyseConfig( @@ -14,7 +14,6 @@ TEST_DIR / "data" / "dcdc" / "charge" / "demo_1.cpp", ], src_dir=TEST_DIR / "data" / "dcdc", - outdir=TEST_DIR / "output", comment_type=123, ), [ @@ -25,7 +24,6 @@ SourceAnalyseConfig( src_files=None, src_dir=TEST_DIR / "data" / "dcdc", - outdir=TEST_DIR / "output", comment_type=123, ), [ @@ -35,8 +33,8 @@ ), ], ) -def test_config_schema_validator_negative(vdocs_config, result): - errors = vdocs_config.check_schema() +def test_config_schema_validator_negative(analyse_config, result): + errors = analyse_config.check_schema() assert sorted(errors) == sorted(result) diff --git a/tests/test_analyse_utils.py b/tests/test_analyse_utils.py index c6c61e8..c78feb0 100644 --- a/tests/test_analyse_utils.py +++ b/tests/test_analyse_utils.py @@ -9,7 +9,7 @@ import tree_sitter_python from sphinx_codelinks.analyse import utils -from sphinx_codelinks.analyse.config import UNIX_NEWLINE +from sphinx_codelinks.config import UNIX_NEWLINE @pytest.fixture(scope="session") diff --git a/tests/test_cmd.py b/tests/test_cmd.py index 0f6e9dc..6d5adbd 100644 --- a/tests/test_cmd.py +++ b/tests/test_cmd.py @@ -5,8 +5,8 @@ import toml from typer.testing import CliRunner -from sphinx_codelinks.analyse.config import CommentType from sphinx_codelinks.cmd import app +from sphinx_codelinks.source_discover.config import CommentType from .conftest import DATA_DIR, TEST_DIR @@ -25,19 +25,32 @@ "exclude": ["**/charge/demo_1.cpp", "**/discharge/demo_3.cpp"], "include": ["**/charge/demo_2.cpp", "**/supercharge.cpp"], "gitignore": True, - "comment_type": [CommentType.cpp.value], + "comment_type": CommentType.cpp.value, } -ANALYSER_CONFIG_TEMPLATE = { - "source_discover": SRC_DISCOVER_TEMPLATE, +ANALYSE_CONFIG_TEMPLATE = { + "get_oneline_needs": True, "oneline_comment_style": ONELINE_COMMENT_TEMPLATE, } +CODELINKS_CONFIG_TEMPLATE = { + "outdir": "set/it/to/somewhere", + "projects": { + "project_1": { + "source_discover": SRC_DISCOVER_TEMPLATE, + "analyse": ANALYSE_CONFIG_TEMPLATE, + } + }, +} runner = CliRunner() @pytest.mark.parametrize( - ("config_path"), [(DATA_DIR / "analyse" / "default_config.toml")] + ("config_path"), + [ + (DATA_DIR / "configs" / "minimum_config.toml"), + (DATA_DIR / "configs" / "full_config.toml"), + ], ) def test_analyse(config_path: Path, tmp_path: Path) -> None: options: list[str] = ["analyse", str(config_path), "--outdir", str(tmp_path)] @@ -73,19 +86,14 @@ def test_discover(options, stdout): @pytest.mark.parametrize( - ("config_dict", "output_lines"), + ("src_discover_dict", "analyse_dict", "output_lines"), [ ( { - key: { - src_key: (123 if src_key == "exclude" else src_value) - for src_key, src_value in value.items() - if isinstance(value, dict) - } - if isinstance(value, dict) and key == "source_discover" - else value - for key, value in ANALYSER_CONFIG_TEMPLATE.items() + src_key: (123 if src_key == "exclude" else src_value) + for src_key, src_value in SRC_DISCOVER_TEMPLATE.items() }, + ANALYSE_CONFIG_TEMPLATE, [ "Invalid value: Invalid source discovery configuration:", "Schema validation error in field 'exclude': 123 is not of type 'array'", @@ -93,14 +101,10 @@ def test_discover(options, stdout): ), ( { - key: { - src_key: (123 if src_key == "include" else src_value) - for src_key, src_value in value.items() - } - if isinstance(value, dict) and key == "source_discover" - else value - for key, value in ANALYSER_CONFIG_TEMPLATE.items() + src_key: (123 if src_key == "include" else src_value) + for src_key, src_value in SRC_DISCOVER_TEMPLATE.items() }, + ANALYSE_CONFIG_TEMPLATE, [ "Invalid value: Invalid source discovery configuration:", "Schema validation error in field 'include': 123 is not of type 'array'", @@ -108,18 +112,12 @@ def test_discover(options, stdout): ), ( { - key: { - src_key: ( - 123 - if src_key in ("exclude", "include", "src_dir") - else src_value - ) - for src_key, src_value in value.items() - } - if isinstance(value, dict) and key == "source_discover" - else value - for key, value in ANALYSER_CONFIG_TEMPLATE.items() + src_key: ( + 123 if src_key in ("exclude", "include", "src_dir") else src_value + ) + for src_key, src_value in SRC_DISCOVER_TEMPLATE.items() }, + ANALYSE_CONFIG_TEMPLATE, [ "Invalid value: Invalid source discovery configuration:", "Schema validation error in field 'src_dir': 123 is not of type 'string'", @@ -128,11 +126,14 @@ def test_discover(options, stdout): ], ), ( + SRC_DISCOVER_TEMPLATE, { - key: ( - {"not_expected": 123} if key == "oneline_comment_style" else value + analyse_key: ( + {"not_expected": 123} + if analyse_key == "oneline_comment_style" + else analyse_value ) - for key, value in ANALYSER_CONFIG_TEMPLATE.items() + for analyse_key, analyse_value in ANALYSE_CONFIG_TEMPLATE.items() }, [ "Invalid value: Invalid oneline comment style configuration:", @@ -141,32 +142,72 @@ def test_discover(options, stdout): ], ), ( + SRC_DISCOVER_TEMPLATE, { - key: ( + analyse_key: ( {"needs_fields": [{"name": "id"}, {"name": "id"}]} - if key == "oneline_comment_style" - else value + if analyse_key == "oneline_comment_style" + else analyse_value ) - for key, value in ANALYSER_CONFIG_TEMPLATE.items() + for analyse_key, analyse_value in ANALYSE_CONFIG_TEMPLATE.items() }, [ - "Invalid value: Invalid oneline comment style configuration:", + "Invalid value: OneLineCommentStyle configuration errors:", "Missing required fields: ['title', 'type']", "Field 'id' is defined multiple times.", ], ), ], ) -def test_analyse_config_negative(config_dict, output_lines, tmp_path: Path) -> None: - # Force disable Rich styling - config_file = tmp_path / "analyse_config.toml" +def test_analyse_config_negative( + src_discover_dict, analyse_dict, output_lines, tmp_path: Path +) -> None: + config_file = tmp_path / "codelinks_config.toml" + codelink_dict = {"codelinks": CODELINKS_CONFIG_TEMPLATE} + codelink_dict["codelinks"]["projects"]["project_1"]["source_discover"] = ( + src_discover_dict + ) + codelink_dict["codelinks"]["projects"]["project_1"]["analyse"] = analyse_dict + with config_file.open("w", encoding="utf-8") as f: + toml.dump(codelink_dict, f) + + options = [ + "analyse", + str(config_file), + ] + result = runner.invoke(app, options) + assert result.exit_code != 0 + for line in output_lines: + assert line in result.stdout + + +@pytest.mark.parametrize( + ("projects", "output_lines"), + [ + ( + ["project_1", "project_2"], + [ + "The following projects are not found:", + "project_2", + ], + ), + ], +) +def test_analyse_project_negative(projects, output_lines, tmp_path: Path) -> None: + config_file = tmp_path / "codelinks_config.toml" + codelink_dict = {"codelinks": CODELINKS_CONFIG_TEMPLATE} with config_file.open("w", encoding="utf-8") as f: - toml.dump(config_dict, f) + toml.dump(codelink_dict, f) + projects_config = [] + for project in projects: + projects_config.append("--project") + projects_config.append(project) options = [ "analyse", str(config_file), ] + options.extend(projects_config) result = runner.invoke(app, options) assert result.exit_code != 0 for line in output_lines: diff --git a/tests/test_oneline_parser.py b/tests/test_oneline_parser.py index ebb2b32..aa4ef91 100644 --- a/tests/test_oneline_parser.py +++ b/tests/test_oneline_parser.py @@ -1,11 +1,11 @@ import pytest -from sphinx_codelinks.analyse.config import ESCAPE, UNIX_NEWLINE, OneLineCommentStyle from sphinx_codelinks.analyse.oneline_parser import ( OnelineParserInvalidWarning, WarningSubTypeEnum, oneline_parser, ) +from sphinx_codelinks.config import ESCAPE, UNIX_NEWLINE, OneLineCommentStyle from .conftest import ONELINE_COMMENT_STYLE, ONELINE_COMMENT_STYLE_DEFAULT diff --git a/tests/test_source_discover.py b/tests/test_source_discover.py index 486fb5e..fac02ad 100644 --- a/tests/test_source_discover.py +++ b/tests/test_source_discover.py @@ -18,7 +18,7 @@ "exclude": ["exclude1", "exclude2"], "include": ["include1", "include2"], "gitignore": True, - "file_types": ["cpp", "hpp"], + "comment_type": "cpp", }, ["Schema validation error in field 'src_dir': 123 is not of type 'string'"], ), @@ -28,7 +28,7 @@ "exclude": ["exclude1", "exclude2"], "include": ["include1", "include2"], "gitignore": "TrueAsString", - "file_types": ["cpp", "hpp"], + "comment_type": "cpp", }, [ "Schema validation error in field 'gitignore': 'TrueAsString' is not of type 'boolean'" @@ -40,10 +40,22 @@ "exclude": ["exclude1", "exclude2"], "include": ["include1", "include2"], "gitignore": True, - "file_types": "py", + "comment_type": "java", }, [ - "Schema validation error in field 'file_types': 'py' is not of type 'array'" + "Schema validation error in field 'comment_type': 'java' is not one of ['cpp', 'python']" + ], + ), + ( + { + "src_dir": "/path/to/root", + "exclude": ["exclude1", "exclude2"], + "include": ["include1", "include2"], + "gitignore": True, + "comment_type": ["cpp", "hpp"], + }, + [ + "Schema validation error in field 'comment_type': ['cpp', 'hpp'] is not of type 'string'" ], ), ], @@ -63,7 +75,14 @@ def test_schema_negative(config, msgs): "exclude": ["exclude1", "exclude2"], "include": ["include1", "include2"], "gitignore": True, - "file_types": ["cpp", "hpp"], + "comment_type": "cpp", + }, + { + "src_dir": "/path/to/root", + "exclude": ["exclude1", "exclude2"], + "include": ["include1", "include2"], + "gitignore": True, + "comment_type": "python", }, ], ) @@ -108,7 +127,7 @@ def test_schema_positive(config): "", ), ( - {"gitignore": False, "file_types": ["cpp"]}, + {"gitignore": False, "comment_type": "cpp"}, 4, "cpp", ), diff --git a/tests/test_src_trace.py b/tests/test_src_trace.py index 8b98287..092cd5e 100644 --- a/tests/test_src_trace.py +++ b/tests/test_src_trace.py @@ -5,17 +5,17 @@ import pytest from sphinx.testing.util import SphinxTestApp -from sphinx_codelinks.analyse.analyse import SourceAnalyse -from sphinx_codelinks.sphinx_extension.config import ( +from sphinx_codelinks.analyse.projects import AnalyseProjects +from sphinx_codelinks.config import ( SRC_TRACE_CACHE, - SrcTraceSphinxConfig, + CodeLinksConfig, check_configuration, ) from sphinx_codelinks.sphinx_extension.source_tracing import set_config_to_sphinx @pytest.mark.parametrize( - ("src_trace_config", "result"), + ("codelinks_config", "result"), [ ( { @@ -25,27 +25,31 @@ "set_remote_url": "TrueString", "projects": { "dcdc": { - "comment_type": "java", - "src_dir": ["../dcdc"], "remote_url_pattern": 44332, - "exclude": [123], - "include": [345], - "gitignore": "_true", - "oneline_comment_style": { - "start_sequence": "[[", - "end_sequence": "]]", - "field_split_char": ",", - "needs_fields": [ - { - "name": "title", - "type": "list[]", - }, - { - "name": "type", - "default": "impl", - "type": "str", - }, - ], + "source_discover": { + "comment_type": "java", + "src_dir": ["../dcdc"], + "exclude": [123], + "include": [345], + "gitignore": "_true", + }, + "analyse": { + "oneline_comment_style": { + "start_sequence": "[[", + "end_sequence": "]]", + "field_split_char": ",", + "needs_fields": [ + { + "name": "title", + "type": "list[]", + }, + { + "name": "type", + "default": "impl", + "type": "str", + }, + ], + }, }, } }, @@ -53,7 +57,7 @@ [ "Project 'dcdc' has the following errors:", "Schema validation error in field 'exclude': 123 is not of type 'string'", - "Schema validation error in field 'file_types': 'java' is not one of ['c', 'cpp', 'h', 'hpp', 'py']", + "Schema validation error in field 'comment_type': 'java' is not one of ['cpp', 'python']", "Schema validation error in field 'gitignore': '_true' is not of type 'boolean'", "Schema validation error in field 'include': 345 is not of type 'string'", "Schema validation error in field 'src_dir': ['../dcdc'] is not of type 'string'", @@ -61,6 +65,7 @@ "Schema validation error in filed 'remote_url_field': 555 is not of type 'string'", "Schema validation error in filed 'set_local_url': 'fdd' is not of type 'boolean'", "Schema validation error in filed 'set_remote_url': 'TrueString' is not of type 'boolean'", + "OneLineCommentStyle configuration errors:", "Schema validation error in need_fields 'title': 'list[]' is not one of ['str', 'list[str]']", "remote_url_pattern must be a string", ], @@ -73,27 +78,31 @@ "set_remote_url": True, "projects": { "dcdc": { - "comment_type": "cpp", - "src_dir": "../dcdc", # intentionally not given "remote_url_pattern": "https://github.com/useblocks/sphinx-codelinks/blob/{commit}/{path}#L{line}", - "exclude": [], - "include": [], - "gitignore": True, - "oneline_comment_style": { - "start_sequence": "[[", - "end_sequence": "]]", - "field_split_char": ",", - "needs_fields": [ - { - "name": "title", - "type": "str", - }, - { - "name": "type", - "default": "impl", - "type": "str", - }, - ], + "source_discover": { + "comment_type": "cpp", + "src_dir": "../dcdc", + "exclude": [], + "include": [], + "gitignore": True, + }, + "analyse": { + "oneline_comment_style": { + "start_sequence": "[[", + "end_sequence": "]]", + "field_split_char": ",", + "needs_fields": [ + { + "name": "title", + "type": "str", + }, + { + "name": "type", + "default": "impl", + "type": "str", + }, + ], + }, }, } }, @@ -107,49 +116,52 @@ ) def test_src_tracing_config_negative( make_app: Callable[..., SphinxTestApp], - src_trace_config, + codelinks_config, result, ): this_file_dir = Path(__file__).parent sphinx_project = Path("data") / "sphinx" app = make_app(srcdir=(this_file_dir / sphinx_project)) - set_config_to_sphinx(src_trace_config, app.env.config) - src_trace_sphinx_config = SrcTraceSphinxConfig(app.env.config) - errors = check_configuration(src_trace_sphinx_config) + set_config_to_sphinx(codelinks_config, app.env.config) + codelinks_sphinx_config = CodeLinksConfig.from_sphinx(app.env.config) + errors = check_configuration(codelinks_sphinx_config) assert sorted(errors) == sorted(result) -def test_src_tracing_config_positive( - make_app: Callable[..., SphinxTestApp], -): - src_trace_config = { +def test_src_tracing_config_positive(make_app: Callable[..., SphinxTestApp], tmp_path): + codelinks_config = { "remote_url_field": "remote-url", "local_url_field": "local-url", "set_local_url": True, "set_remote_url": True, + "outdir": tmp_path, "projects": { "dcdc": { - "comment_type": "cpp", - "src_dir": "../dcdc", + "source_discover": { + "comment_type": "cpp", + "src_dir": "../dcdc", + "exclude": ["**/*.hpp"], + "include": ["**/*.cpp"], + "gitignore": True, + }, "remote_url_pattern": "https://github.com/useblocks/sphinx-codelinks/blob/{commit}/{path}#L{line}", - "exclude": ["**/*.hpp"], - "include": ["**/*.cpp"], - "gitignore": True, - "oneline_comment_style": { - "start_sequence": "[[", - "end_sequence": "]]", - "field_split_char": ",", - "needs_fields": [ - { - "name": "title", - "type": "str", - }, - { - "name": "type", - "default": "impl", - "type": "str", - }, - ], + "analyse": { + "oneline_comment_style": { + "start_sequence": "[[", + "end_sequence": "]]", + "field_split_char": ",", + "needs_fields": [ + { + "name": "title", + "type": "str", + }, + { + "name": "type", + "default": "impl", + "type": "str", + }, + ], + }, }, } }, @@ -157,9 +169,9 @@ def test_src_tracing_config_positive( this_file_dir = Path(__file__).parent sphinx_project = Path("data") / "sphinx" app = make_app(srcdir=(this_file_dir / sphinx_project)) - set_config_to_sphinx(src_trace_config, app.env.config) - src_trace_sphinx_config = SrcTraceSphinxConfig(app.env.config) - errors = check_configuration(src_trace_sphinx_config) + set_config_to_sphinx(codelinks_config, app.env.config) + codelinks_sphinx_config = CodeLinksConfig.from_sphinx(app.env.config) + errors = check_configuration(codelinks_sphinx_config) assert not errors @@ -207,7 +219,7 @@ def test_build_html( html = Path(app.outdir, "index.html").read_text() assert html - warnings = SourceAnalyse.load_warnings(Path(app.outdir) / SRC_TRACE_CACHE) + warnings = AnalyseProjects.load_warnings(Path(app.outdir) / SRC_TRACE_CACHE) assert not warnings assert app.env.get_doctree("index") == snapshot_doctree