Feature/external sources in needsjson#1740
Open
cpolzer wants to merge 4 commits into
Open
Conversation
added 4 commits
July 6, 2026 15:51
When external needs are included in the needs.json output (via needs_builder_filter), the external source configuration metadata is now serialized into an 'external_sources' section in the version block. Each external need also gets an 'external_source' field pointing to the base_url of its source. This enables downstream consumers to reconstruct the full provenance chain when re-exporting external needs transitively: - Direct sources have origin=null - Inherited (transitive) sources record which intermediate project re-exported them via the 'origin' field Changes: - data.py: Add ExternalSourceInfo TypedDict, external_source_map and inherited_external_sources to SphinxNeedsData, merge support - external_needs.py: Register source mapping per need, inherit external_sources from consumed JSON files - needsfile.py: Serialize external_sources and per-need external_source into the needs.json output - tests: Add provenance chain round-trip test (X -> A -> B)
…pshots - Apply ruff format to data.py and external_needs.py - Fix RST heading level inconsistency in configuration.rst - Update snapshots for test_link_conditions, test_need_constraints, and test_needs_from_toml (these projects also use external needs)
The json_path field in external_sources contains platform-specific absolute paths (/private/var/folders/... on macOS vs /tmp/... on Linux), causing snapshot test failures in CI. This commit adds a helper function to remove json_path from the test data before snapshot comparison, similar to how we exclude 'created', 'project', etc. The json_path value is an implementation detail and not critical to verify in snapshot tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When needs_builder_filter is relaxed to include external needs in needs.json, downstream consumers lose the provenance metadata. (information on the configured external needs in a producer -> links can for consumers not be generated anymore).
This probably is not an easy thing to merge - i would guess other tooling handling external needs would need to adapt first or follow?
The extended example is:
1.) I have a sphinx-needs "Project A" consuming external needs and publishing them to needs.json
2.) "Project B" consumes "Project A" needs.json -> the links from project A cannot be rebuild because the external needs configuration of "Project A" is not known.
Solution
Testing
tox -e py312 -- tests/test_external.py tests/test_needs_external_needs_build.py