From 831c5f0bec7ca45e54bf7474e029135182dfb49e Mon Sep 17 00:00:00 2001 From: "jui-wen.chen" Date: Mon, 11 Aug 2025 22:06:48 +0200 Subject: [PATCH 1/8] versioned and update docs --- README.md | 4 +-- docs/source/development/change_log.rst | 36 +++++++++++++++++++-- docs/source/development/roadmap.rst | 43 ++++++++++++++------------ pyproject.toml | 2 +- 4 files changed, 60 insertions(+), 25 deletions(-) diff --git a/README.md b/README.md index 7e28f34..e39bbbb 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ A Sphinx extension for discovering, linking, and documenting source code across ## Features - **Source Discovery**: Automatically discover source files in your project -- **Virtual Documentation**: Generate documentation from code without modifying source files +- **Analyse**: Parse source codes and extract specified markers with their metadata - **Code Linking**: Create intelligent links between code elements - **Sphinx Integration**: Seamless integration with existing Sphinx documentation @@ -28,7 +28,7 @@ Full documentation: https://codelinks.useblocks.com ## Components - **Source Discovery** ([`src/sphinx_codelinks/source_discover`](src/sphinx_codelinks/source_discover)): Code analysis and discovery -- **Virtual Docs** ([`src/sphinx_codelinks/virtual_docs`](src/sphinx_codelinks/virtual_docs)): Documentation generation +- **Analyse** ([`src/sphinx_codelinks/analyse`](src/sphinx_codelinks/analyse)): Documentation generation - **Sphinx Extension** ([`src/sphinx_codelinks/sphinx_extension`](src/sphinx_codelinks/sphinx_extension)): Sphinx integration - **Command Line** ([`src/sphinx_codelinks/cmd.py`](src/sphinx_codelinks/cmd.py)): CLI interface diff --git a/docs/source/development/change_log.rst b/docs/source/development/change_log.rst index 6d38699..1d1c059 100644 --- a/docs/source/development/change_log.rst +++ b/docs/source/development/change_log.rst @@ -3,7 +3,39 @@ Changelog ========= -.. _release:0.1.2: +.. _`release:1.0.0`: + +1.0.0 +----- + +:Released: 12.08.2025 + +New and Improved +................ + +- ✨ Added a new ``analyse`` CLI command and corresponding API. + + The ``analyse`` command parses source files (Python, C/C++) and extracts markers from comments. + It can extract three types of markers, as documented in the :ref:`analyse ` section: + + - One-line need definitions + - Need ID references + - Marked RST blocks + + The extracted markers and their metadata are saved to a JSON file for further processing. + +- 🔨 Replaced ``virtual_docs`` with the new ``analyse`` module. + + The ``virtual_docs`` feature, which handled one-line need definitions (:ref:`OneLineCommentStyle `), + has been migrated into the new ``analyse`` module and removed from the core. + The caching feature of ``virtual_docs`` is temporarily removed and may be reintroduced later. + +- 🔨 Updated the ``src-trace`` Sphinx directive. + + The ``src-trace`` directive now uses the new ``analyse`` API instead of the old ``virtual_docs`` one. + Note that the configuration files for ``src-trace`` and the ``analyse`` CLI are not yet compatible; this will be addressed in a future release. + +.. _`release:0.1.2`: 0.1.2 ----- @@ -24,7 +56,7 @@ Fixes Local links between docs and one-line need definitions work correctly, when :ref:`src_dir ` in multiple project configurations point at different locations. -.. _release:0.1.1: +.. _`release:0.1.1`: 0.1.1 ----- diff --git a/docs/source/development/roadmap.rst b/docs/source/development/roadmap.rst index b21529a..fcd8d83 100644 --- a/docs/source/development/roadmap.rst +++ b/docs/source/development/roadmap.rst @@ -3,31 +3,34 @@ Roadmap ======= -Other Comment styles --------------------- +Command-Line Interface +---------------------- -Currently, only ``C/C++`` comment style is supported. -The other comment styles for different programming languages are planed, such as: +- Introduce configurable ``--verbose`` and ``--quiet`` logging options. -- Python -- Rust -- YAML -- SyML +Configuration Files +------------------- -Nested .gitignore ------------------ +- Unify TOML configuration for the ``src-trace`` directive and the ``analyse`` CLI. +- Support automatic discovery of TOML configuration files (e.g., ``pyproject.toml``). +- Discourage global configuration in TOML files to promote project-specific settings. +- Improve integration with the ``ubCode`` and ``Sphinx-Needs`` ecosystems. -``CodeLinks`` respects ``.gitignore`` file, but if the .gitignore files are nested, it's not supported. -Respecting nested ``.gitignore`` in the context of the git repositories is planned. +Source Code Parsing +------------------- -Flexible way to define Sphinx-Needs need items in source code -------------------------------------------------------------- +- Introduce a configurable option to strip leading characters (e.g., ``*``) from commented RST blocks. +- Enrich tagged scopes with additional metadata. +- Extend language support by adding parsers for more comment styles, including: -The only way to define ``Sphinx-Needs`` need items is through ``one-line comment style``. -Raw RST text and multi-lines comments style are planned to support + - Rust + - YAML + - SyML -Export needs.json ------------------ +- Enhance ``.gitignore`` handling to support nested ``.gitignore`` files. -To facilitate CI workflow and enhance the portability of ``need items`` defined in source code, -we plan to have the feature to export the needs defined in source code to a JSON file. +Defining Needs in Source Code +----------------------------- + +- Introduce flexible ways to define ``Sphinx-Needs`` items in source code, such as using raw RST text and multi-line comments. +- Implement a feature to export needs defined in source code to a ``needs.json`` file, improving CI workflows and portability. diff --git a/pyproject.toml b/pyproject.toml index 67040ff..86018a4 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "sphinx-codelinks" -version = "0.1.2" +version = "1.0.0" description = "Fast Source Code Traceability for Sphinx-Needs" authors = [{ name = "team useblocks", email = "info@useblocks.com" }] maintainers = [ From 913d587bb705dcf25593b97a88ea6ba87791c6c8 Mon Sep 17 00:00:00 2001 From: juiwenchen Date: Tue, 12 Aug 2025 10:08:41 +0200 Subject: [PATCH 2/8] updated the abstract --- docs/source/index.rst | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/source/index.rst b/docs/source/index.rst index f1bcc25..fd634af 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -12,9 +12,11 @@ .. div:: sd-fs-5 sd-font-italic - ``Sphinx-CodeLinks`` is designed for Engineering-as-Code workflows to facilitate Application Lifecycle Management (ALM). - It enables users to define ``Sphinx-Needs`` need items within source code using a single line and automatically extract them - into the documentation during the Sphinx build process. + ``Sphinx-CodeLinks`` is a lightweight solution for enabling traceability between source code and documentation. + With a single line in your code, you can: + + 1. Defining new ``Sphinx-Needs`` - extracted and integrated into the Sphinx build + 2. Embedding existing need IDs - extracted for cross-referencing with need items in the documentation .. grid:: 1 1 2 2 :gutter: 2 2 3 3 From 99b1f81752ae7a33e138c91bd1d7d79679949f51 Mon Sep 17 00:00:00 2001 From: juiwenchen Date: Tue, 12 Aug 2025 10:39:22 +0200 Subject: [PATCH 3/8] updated dcos --- docs/source/components/directive.rst | 4 ++- docs/source/components/oneline.rst | 54 +++++++++++++++++++--------- 2 files changed, 40 insertions(+), 18 deletions(-) diff --git a/docs/source/components/directive.rst b/docs/source/components/directive.rst index 4012aa5..64f6395 100644 --- a/docs/source/components/directive.rst +++ b/docs/source/components/directive.rst @@ -3,7 +3,9 @@ Directive ========= -``CodeLinks`` provides the ``src-trace`` directive and it can be used in the following ways: +.. attention:: ``src-trace`` directive currently only supports :ref:`one-line need definition `. + +``CodeLinks`` provides ``src-trace`` directive and it can be used in the following ways: .. code-block:: rst diff --git a/docs/source/components/oneline.rst b/docs/source/components/oneline.rst index 63ee49b..8b0c470 100644 --- a/docs/source/components/oneline.rst +++ b/docs/source/components/oneline.rst @@ -91,14 +91,25 @@ For example, with the following **needs_fields** configuration: .. _`fields_config`: -.. code-block:: python +.. tabs:: + + .. code-tab:: python + + needs_fields = [ + {"name": "title"}, + {"name": "id"}, + {"name": "type", "default": "impl"}, + {"name": "links", "type": "list[str]", "default": []}, + ], + + .. code-tab:: toml - needs_fields=[ - {"name": "title"}, - {"name": "id"}, - {"name": "type", "default": "impl"}, - {"name": "links", "type": "list[str]", "default": []}, - ], + needs_fields = [ + { name = "title" }, + { name = "id" }, + { name = "type", default = "impl" }, + { name = "links", type = "list[str]", default = [] }, + ] the one-line comment shall be defined as follows: @@ -122,17 +133,26 @@ when it is not given in the need definition. For example, with the following needs_fields definition, -.. code-block:: python +.. tabs:: + + .. code-tab:: python + + needs_fields = [ + { + "name": "title" + }, + { + "name": "type", + "default": "implementation" + }, + ] + + .. code-tab:: toml - needs_fields = [ - { - "name": "title" - }, - { - "name": "type", - "default": "implementation" - }, - ] + needs_fields = [ + { name = "title" }, + { name = "type", default = "implementation" } + ] the following need definition in source code is equivalent to RST shown below: From 28147454ec9751378ad5128a292f01d7bc229b14 Mon Sep 17 00:00:00 2001 From: juiwenchen Date: Fri, 22 Aug 2025 13:37:14 +0200 Subject: [PATCH 4/8] updated docs --- docs/source/components/write.rst | 49 ++++++++++++++++++++++++++ docs/source/development/change_log.rst | 14 ++++++-- docs/source/development/roadmap.rst | 1 - docs/source/index.rst | 1 + 4 files changed, 62 insertions(+), 3 deletions(-) create mode 100644 docs/source/components/write.rst diff --git a/docs/source/components/write.rst b/docs/source/components/write.rst new file mode 100644 index 0000000..9245d78 --- /dev/null +++ b/docs/source/components/write.rst @@ -0,0 +1,49 @@ +.. _write: + +Write +===== + +The ``write`` command is used to generate reStructuredText files from the markers extracted by the ``analyse`` command. +This allows you to create documentation that includes links to source code based on the need ids specified in your code comments. + +Example usage +------------- + +With the following extracted markers fron the ``analyse`` command, + +.. code-block:: json + + { + "my_project": [ + { + "filepath": "/home/demo/git_repo/ub/sphinx-codelinks/tests/data/need_id_refs/dummy_1.cpp", + "remote_url": "https://github.com/useblocks/sphinx-codelinks/blob/main/tests/data/need_id_refs/dummy_1.cpp#L3", + "source_map": { + "start": {"row": 2, "column": 13}, + "end": {"row": 2, "column": 51} + }, + "tagged_scope": "void dummy_func1(){\n //...\n }", + "need_ids": ["NEED_001", "NEED_002", "NEED_003", "NEED_004"], + "marker": "@need-ids:", + "type": "need-id-refs" + }, + ], + } + +The following RST file with :external+needs:ref:`needextend ` directive can be generated by the ``write rst`` command: + +.. code-block:: rst + + .. needextend:: NEED_001 + :remote-url: https://github.com/useblocks/sphinx-codelinks/blob/main/tests/data/need_id_refs/dummy_1.cpp#L3 + + .. needextend:: NEED_002 + :remote-url: https://github.com/useblocks/sphinx-codelinks/blob/main/tests/data/need_id_refs/dummy_1.cpp#L3 + + .. needextend:: NEED_003 + :remote-url: https://github.com/useblocks/sphinx-codelinks/blob/main/tests/data/need_id_refs/dummy_1.cpp#L3 + + .. needextend:: NEED_004 + :remote-url: https://github.com/useblocks/sphinx-codelinks/blob/main/tests/data/need_id_refs/dummy_1.cpp#L3 + +More examples can be found in `test cases `__ diff --git a/docs/source/development/change_log.rst b/docs/source/development/change_log.rst index 1d1c059..f448902 100644 --- a/docs/source/development/change_log.rst +++ b/docs/source/development/change_log.rst @@ -8,7 +8,7 @@ Changelog 1.0.0 ----- -:Released: 12.08.2025 +:Released: 22.08.2025 New and Improved ................ @@ -24,6 +24,11 @@ New and Improved The extracted markers and their metadata are saved to a JSON file for further processing. +- ✨ Added a new ``write rst`` CLI command. + + The ``write rst`` command write a reStructuredText file with :external+needs:ref:`needextend ` directive from the extracted markers generated by ``analyse``. + The generated RST can be included in the Sphinx documentation to create the source code links in the existing needs + - 🔨 Replaced ``virtual_docs`` with the new ``analyse`` module. The ``virtual_docs`` feature, which handled one-line need definitions (:ref:`OneLineCommentStyle `), @@ -33,7 +38,12 @@ New and Improved - 🔨 Updated the ``src-trace`` Sphinx directive. The ``src-trace`` directive now uses the new ``analyse`` API instead of the old ``virtual_docs`` one. - Note that the configuration files for ``src-trace`` and the ``analyse`` CLI are not yet compatible; this will be addressed in a future release. + +- 🔨 Unified configuration in TOML + + The configuration for ``src-trace`` directive defined in TOML is now compatible with the new ``analyse`` module. + + .. _`release:0.1.2`: diff --git a/docs/source/development/roadmap.rst b/docs/source/development/roadmap.rst index fcd8d83..706fbf0 100644 --- a/docs/source/development/roadmap.rst +++ b/docs/source/development/roadmap.rst @@ -25,7 +25,6 @@ Source Code Parsing - Rust - YAML - - SyML - Enhance ``.gitignore`` handling to support nested ``.gitignore`` files. diff --git a/docs/source/index.rst b/docs/source/index.rst index fd634af..94ff269 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -65,6 +65,7 @@ Contents components/oneline components/analyse components/discover + components/write .. toctree:: :maxdepth: 1 From 2dbc90bca7cfe71e6b91e18ec2e6a12ac025779b Mon Sep 17 00:00:00 2001 From: juiwenchen Date: Fri, 22 Aug 2025 13:54:00 +0200 Subject: [PATCH 5/8] updated docs --- README.md | 1 + docs/source/components/configuration.rst | 2 +- src/sphinx_codelinks/README.md | 46 ------------------------ 3 files changed, 2 insertions(+), 47 deletions(-) delete mode 100644 src/sphinx_codelinks/README.md diff --git a/README.md b/README.md index e39bbbb..002db0b 100644 --- a/README.md +++ b/README.md @@ -19,6 +19,7 @@ Add to your `conf.py`: ```python extensions = ['sphinx_needs', 'sphinx_codelinks'] +src_trace_config_from_toml = "codelinks.toml" ``` ## Documentation diff --git a/docs/source/components/configuration.rst b/docs/source/components/configuration.rst index b078bf1..09b7713 100644 --- a/docs/source/components/configuration.rst +++ b/docs/source/components/configuration.rst @@ -28,7 +28,7 @@ Specifies the path to a `TOML file `__ containing **Sphinx-Code .. code-block:: python # In conf.py - src_trace_config_from_toml = "src_trace.toml" + src_trace_config_from_toml = "codelinks.toml" When using a TOML configuration file: diff --git a/src/sphinx_codelinks/README.md b/src/sphinx_codelinks/README.md deleted file mode 100644 index de5c307..0000000 --- a/src/sphinx_codelinks/README.md +++ /dev/null @@ -1,46 +0,0 @@ -# CodeLinks - -## Overview - -This is a Sphinx extension to extract Sphinx-Needs items from source files -such as C, C++ and others. - -The need items are defined in the source files as comments and can be used to define -test case specifications or implementation markers. - -Various definition styles are supported, such as one-line, multi-line or raw RST. - -The project consists of the following three components: - -- Source Discovery: determines list of source files from a given directory -- Virtual Docs: extract need annotations while keeping the source map -- Source Tracing: Sphinx extension to represent the collected the needs in the documentation - -`Source Discovery` and `Virtual Docs` can be used as `APIs` or `CLI tools`. -The detail usages can be found in the [test cases](./tests). - -The library is built to be - -- ⚡ fast for large code bases and -- 📃 support a multitude of languages. - -## Source Discovery - -Recursively collect the file paths from a given directory. -It can be configured to respect `.gitignore`. - -## Virtual Docs - -Virtual Docs parses the discoverd files and - -- extracts the need items from the comments in the source files. -- extracts additional metadata such as extra options and links. -- generates virtual documents containing the above-mentioned information into `json` files. -- caches virtual docs for incremental builds. -- keeps the source map to the path and line number of the original source files. - -## CodeLinks - -CodeLinks is a Sphinx Extension based on Sphinx-Needs. It provides the directive `src-tracing` -to collect the needs defined in source files by using `Source Discovery` and `Virtual Docs` -under the hood. From cc73a6295071a0ab74d0c1ce17720eb903781b7b Mon Sep 17 00:00:00 2001 From: juiwenchen <46341773+juiwenchen@users.noreply.github.com> Date: Fri, 22 Aug 2025 14:04:21 +0200 Subject: [PATCH 6/8] Update docs/source/components/write.rst Co-authored-by: Marco Heinemann --- docs/source/components/write.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/components/write.rst b/docs/source/components/write.rst index 9245d78..565cfcf 100644 --- a/docs/source/components/write.rst +++ b/docs/source/components/write.rst @@ -3,7 +3,7 @@ Write ===== -The ``write`` command is used to generate reStructuredText files from the markers extracted by the ``analyse`` command. +The ``write`` command is used to generate file formats such as reStructuredText from the markers extracted by the ``analyse`` command. This allows you to create documentation that includes links to source code based on the need ids specified in your code comments. Example usage From d646b3e11eb713bb7de526ee5d0fca3209e44842 Mon Sep 17 00:00:00 2001 From: juiwenchen <46341773+juiwenchen@users.noreply.github.com> Date: Fri, 22 Aug 2025 14:04:38 +0200 Subject: [PATCH 7/8] Update docs/source/development/roadmap.rst Co-authored-by: Marco Heinemann --- docs/source/development/roadmap.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/development/roadmap.rst b/docs/source/development/roadmap.rst index 706fbf0..a7da5a7 100644 --- a/docs/source/development/roadmap.rst +++ b/docs/source/development/roadmap.rst @@ -21,7 +21,7 @@ Source Code Parsing - Introduce a configurable option to strip leading characters (e.g., ``*``) from commented RST blocks. - Enrich tagged scopes with additional metadata. -- Extend language support by adding parsers for more comment styles, including: +- Extend language support by adding parsers for more comment styles, including but not limited to: - Rust - YAML From 57c5c024aea4889c6d8594b99fe10dd5dfa29f1c Mon Sep 17 00:00:00 2001 From: juiwenchen Date: Fri, 22 Aug 2025 14:09:47 +0200 Subject: [PATCH 8/8] updated according to the review --- docs/source/components/write.rst | 10 +++++----- docs/source/development/change_log.rst | 2 -- docs/source/development/roadmap.rst | 2 -- 3 files changed, 5 insertions(+), 9 deletions(-) diff --git a/docs/source/components/write.rst b/docs/source/components/write.rst index 565cfcf..78d9ea3 100644 --- a/docs/source/components/write.rst +++ b/docs/source/components/write.rst @@ -17,7 +17,7 @@ With the following extracted markers fron the ``analyse`` command, "my_project": [ { "filepath": "/home/demo/git_repo/ub/sphinx-codelinks/tests/data/need_id_refs/dummy_1.cpp", - "remote_url": "https://github.com/useblocks/sphinx-codelinks/blob/main/tests/data/need_id_refs/dummy_1.cpp#L3", + "remote_url": "https://github.com/useblocks/sphinx-codelinks/blob/951e40e7845f06d5cfc4ca20ebb984308fdaf985/tests/data/need_id_refs/dummy_1.cpp#L3", "source_map": { "start": {"row": 2, "column": 13}, "end": {"row": 2, "column": 51} @@ -35,15 +35,15 @@ The following RST file with :external+needs:ref:`needextend ` direct .. code-block:: rst .. needextend:: NEED_001 - :remote-url: https://github.com/useblocks/sphinx-codelinks/blob/main/tests/data/need_id_refs/dummy_1.cpp#L3 + :remote-url: https://github.com/useblocks/sphinx-codelinks/blob/951e40e7845f06d5cfc4ca20ebb984308fdaf985/tests/data/need_id_refs/dummy_1.cpp#L3 .. needextend:: NEED_002 - :remote-url: https://github.com/useblocks/sphinx-codelinks/blob/main/tests/data/need_id_refs/dummy_1.cpp#L3 + :remote-url: https://github.com/useblocks/sphinx-codelinks/blob/951e40e7845f06d5cfc4ca20ebb984308fdaf985/tests/data/need_id_refs/dummy_1.cpp#L3 .. needextend:: NEED_003 - :remote-url: https://github.com/useblocks/sphinx-codelinks/blob/main/tests/data/need_id_refs/dummy_1.cpp#L3 + :remote-url: https://github.com/useblocks/sphinx-codelinks/blob/951e40e7845f06d5cfc4ca20ebb984308fdaf985/tests/data/need_id_refs/dummy_1.cpp#L3 .. needextend:: NEED_004 - :remote-url: https://github.com/useblocks/sphinx-codelinks/blob/main/tests/data/need_id_refs/dummy_1.cpp#L3 + :remote-url: https://github.com/useblocks/sphinx-codelinks/blob/951e40e7845f06d5cfc4ca20ebb984308fdaf985/tests/data/need_id_refs/dummy_1.cpp#L3 More examples can be found in `test cases `__ diff --git a/docs/source/development/change_log.rst b/docs/source/development/change_log.rst index f448902..a943dc7 100644 --- a/docs/source/development/change_log.rst +++ b/docs/source/development/change_log.rst @@ -43,8 +43,6 @@ New and Improved The configuration for ``src-trace`` directive defined in TOML is now compatible with the new ``analyse`` module. - - .. _`release:0.1.2`: 0.1.2 diff --git a/docs/source/development/roadmap.rst b/docs/source/development/roadmap.rst index a7da5a7..b193139 100644 --- a/docs/source/development/roadmap.rst +++ b/docs/source/development/roadmap.rst @@ -11,9 +11,7 @@ Command-Line Interface Configuration Files ------------------- -- Unify TOML configuration for the ``src-trace`` directive and the ``analyse`` CLI. - Support automatic discovery of TOML configuration files (e.g., ``pyproject.toml``). -- Discourage global configuration in TOML files to promote project-specific settings. - Improve integration with the ``ubCode`` and ``Sphinx-Needs`` ecosystems. Source Code Parsing