✨ Add needs_variant_data — rich structured variant data (deprecates needs_filter_data) (#1715)

## Summary

Introduces `needs_variant_data` (inline nested dict) and
`needs_variant_data_file` (path to JSON file) as new configuration
options, providing a namespaced, structured replacement for the flat
`needs_filter_data`. In filter expressions, the data is accessible under
the `var` namespace:

```rst
.. needtable::
   :filter: var.cpu == "arm" and var.build.debug == True

.. needtable::
   :filter: "feature_a" in var.build.features
```

`needs_filter_data` is deprecated with a warning pointing users to the
new option.

## Motivation

- `needs_filter_data` is flat (`dict[str, str]`) and injects keys at
root level, risking collisions with need field names.
- Users working with kconfig-like systems need nested structured data
(e.g. `{"build": {"arch": "arm", "debug": true}}`).
- A dedicated `var` namespace avoids collisions and has been developed
for compatibility with the ubcode query engine which already parses
`var.a.b` attribute chains natively.

## Design Decisions

| Decision | Choice |
|----------|--------|
| Namespace | `var` (e.g. `var.cpu == "arm"`) |
| Config style | Single slot + file override (no named profiles) | |
Switching | `sphinx-build -D needs_variant_data_file=other.json` | |
`filter_data` | Deprecated with warning, keeps working | | Leaf types |
`str`, `bool`, `int`, `float` |
| Arrays | Uniform scalar type only |
| Precedence | File loaded first, inline deep-merged on top (inline
wins) | | Attribute access | Dot-only (`var.a.b`), no item access | |
Missing key | Strict `AttributeError` (catches typos in filters) |

## Alternatives Considered

### Named profiles (rejected)

We considered letting users define multiple named variant configurations
upfront and selecting between them:

```toml
[needs]
selected_variant = "arm_debug"

[needs.variant_profiles.arm_debug]
file = "variants/arm_debug.json"
```

**Why rejected:**
- Adds config complexity without clear benefit over just switching the
file path directly.
- Users with multiple profiles already manage them as separate files;
they don't need an indirection layer in sphinx config.
- The simpler "just point at a file" approach composes better with
external tooling (CI matrices, kconfig generators, etc.) that already
produce a single output file.
- Can always be added later as a convenience layer on top if demand
materializes.

### Namespace name

Also considered:
- `env` — too easily confused with environment variables or Sphinx
`BuildEnvironment`
- `cfg` — Rust-flavored, less obvious to Python/RST users
- No namespace (keep flat like `filter_data`) — loses the
collision-safety motivation entirely

`var` was chosen as short, unambiguous, and evocative of "variant" /
"variable".

### `SimpleNamespace` via `json.loads` object_hook (rejected)

Quick and elegant for prototyping, but rejected for production because:
- No validation of leaf types or uniform arrays
- No controlled error messages for missing keys (gives generic
`AttributeError`)
- Exposes internal `SimpleNamespace` attributes (`__dict__`, etc.) to
eval context
- No way to add `__contains__` for `"x" in var.tags` support without
subclassing

### `__getitem__` fallback (rejected)

Considered allowing both `var.a.b` and `var["a"]["b"]`. Rejected
because:
- Dot-access is significantly more readable in filter expressions:
`var.build.debug` vs `var["build"]["debug"]`
- Consistent with Jinja2 template syntax where `foo.bar` and
`foo["bar"]` are equivalent — users of Sphinx/Jinja are already familiar
with dot-access for nested data
- Keeps the API surface minimal and forces a single idiomatic style
- Can be added later if needed

## User-facing Configuration

```python
# conf.py — inline (simple cases)
needs_variant_data = {
    "cpu": "arm",
    "debug": True,
    "build": {"optimization": 2, "features": ["feature_a", "feature_b"]},
}

# Or point to a file (kconfig-generated JSON, etc.)
needs_variant_data_file = "variants/arm_debug.json"

# Both can be combined: file loaded first, inline overrides on top
```

Switch at build time: `sphinx-build -D
needs_variant_data_file=variants/x86_release.json docs/ _build/`

### Allowed Data Shape

```json
{
  "cpu": "arm",
  "debug": true,
  "build": {
    "optimization": 2,
    "features": ["feature_a", "feature_b"]
  }
}
```

Constraints:

- Top-level must be a dict
- Intermediate nodes: dicts only
- Leaf values: `str | bool | int | float | list[str] | list[bool] |
list[int] | list[float]`
- Arrays must be uniform (all elements same scalar type)
- No `null` values, no dicts inside arrays

## Implementation

- **`sphinx_needs/variant_data.py`** (new) — `VariantDataProxy`,
validation, JSON file loading, deep-merge
- **`sphinx_needs/config.py`** — Two new fields: `variant_data` and
`variant_data_file`
- **`sphinx_needs/needs.py`** — Resolution in `prepare_env` (load file,
merge, validate, store); deprecation warning in `check_configuration`
- **`sphinx_needs/filter_common.py`** — Injects `var` proxy at all
filter eval sites (including `filter_single_need`, `filter_import_item`,
`apply_default_predicate`)
- **`sphinx_needs/functions/functions.py`** — Injects `var` in variant
expression context
- **`sphinx_needs/ubquery.py`** — AST fast-path support for `var.*`
attribute chains (comparisons, `in`/`not in`), avoiding `eval()`
overhead

### Fast-path details (`ubquery.py`)

The AST-based fast-path (`try_build_simple_predicate`) compiles simple
filter expressions to native Python lambdas without `eval()`. This PR
extends it to handle `var.*` attribute chains:

1. `_unpack_attribute_chain()` — unpacks `var.build.debug` AST into
`("var", "build", "debug")`
2. `_resolve_chain()` — traverses the fallback context to resolve the
chain at eval time
3. Comparison handlers recognize `ast.Attribute` chains rooted in
`_FALLBACK_ROOTS` (`{"var"}`)
4. `"value" in var.field` membership tests are also compiled natively
5. Bare `var` as an `ast.Name` bails to the slow path (added to
`_CONTEXT_ONLY_NAMES`)

## Documentation

- **`docs/configuration.rst`** — New `needs_variant_data` and
`needs_variant_data_file` sections with full usage examples;
`needs_filter_data` moved to "Deprecated Options" with a `..
deprecated:: 8.2.0` directive pointing to the new options
- **`docs/directives/need.rst`** — Updated `jinja_content` context
reference
- **`docs/dynamic_functions.rst`** — Updated variant evaluation context
reference
- All predicates context lists (under `needs_fields`, `needs_links`,
`needs_global_options`) now reference `needs_variant_data`

## Tests

- **`tests/test_variant_data.py`** — Unit tests for `VariantDataProxy`,
validation, deep-merge, file loading, fast-path compilation (28 tests)
- **`tests/test_variant_data_integration.py`** — Integration tests with
Sphinx builds: inline config with `needs_warnings`, file loading with
inline override (2 tests)
- **`tests/test_ubquery.py`** — `TestVariantDataFastPath` class:
comparisons, nested access, membership, compound expressions, error
cases (23 new tests)
- Updated existing tests (`test_needs_filter_data`, `test_variants`,
`test_parallel_execution`) to expect the new deprecation warning

## Notes & Edge Cases

- **`-D` override granularity**: Sphinx `-D` cannot set nested dict keys
individually. Switching is done by pointing at a different JSON file.
- **Array uniformity**: Validated at config-load time (fail fast with
clear error).
- **`eval()` safety**: `VariantDataProxy` has no `__getitem__`,
`__setattr__`, or `__delattr__` — only `__getattr__` for reads. Cannot
mutate data from a filter.
- **Relative paths**: `needs_variant_data_file` is resolved relative to
the Sphinx `confdir`.

## Future Improvements

- **Include variant data in `needs.json`**: The resolved variant data
could be emitted in the `needs.json` metadata section, enabling
downstream tools (e.g. ubcode) to know which variant configuration was
active during the build. This would support diffing outputs across
variants and reproducing builds.

Author: chrisjsewell

docs/configuration.rst

@@ -342,7 +342,7 @@ For ``predicates``, the match expression is a string, using Python syntax, that
 - ``is_import`` (``bool``)
 - :ref:`needs_fields`
 - :ref:`needs_links` (``tuple[str, ...]``)
-- :ref:`needs_filter_data`
+- :ref:`needs_variant_data` (via the ``var`` namespace)
 
 For example:
 
@@ -454,7 +454,7 @@ For ``predicates``, the match expression is a string, using Python syntax, that
 - ``is_import`` (``bool``)
 - :ref:`needs_fields`
 - :ref:`needs_links` (``tuple[str, ...]``)
-- :ref:`needs_filter_data`
+- :ref:`needs_variant_data` (via the ``var`` namespace)
 
 For example:
 
@@ -570,55 +570,101 @@ Use ``style_start`` and ``style_end`` like this:
    and orientation (``left``, ``rigth``, ``up`` and ``down``). We suggest to set the orientation
    in ``style_end`` like in the example above, as this is more often supported.
 
-.. _`needs_filter_data`:
+.. _`needs_variant_data`:
 
-needs_filter_data
-~~~~~~~~~~~~~~~~~
+needs_variant_data
+~~~~~~~~~~~~~~~~~~
 
-This option allows to use custom data inside a :ref:`filter_string`.
+.. versionadded:: 8.2.0
+
+Provides structured variant data accessible under the ``var`` namespace in :ref:`filter expressions <filter_string>`.
+This replaces the flat :ref:`needs_filter_data` option with a namespaced, nested data structure
+that avoids collisions with need field names.
+
+The dot-access syntax (``var.build.debug``) is more readable than dictionary-style access (``var["build"]["debug"]``)
+and mirrors `Jinja2 template syntax <https://jinja.palletsprojects.com/en/latest/templates/#variables>`__,
+which Sphinx users are already familiar with.
 
 Configuration example:
 
 .. code-block:: python
 
-   def custom_defined_func():
-       return "my_tag"
-
-   needs_filter_data = {
-       "current_variant": "project_x",
-       "sphinx_tag": custom_defined_func(),
+   needs_variant_data = {
+       "cpu": "arm",
+       "debug": True,
+       "build": {
+           "optimization": 2,
+           "features": ["feature_a", "feature_b"],
+       },
    }
 
-The defined ``needs_filter_data`` must be a dictionary. Its values can be a string variable or a custom defined
-function. The function get executed during config loading and must return a string.
+The data is then available in filter expressions via the ``var`` namespace:
 
-The value of ``needs_filter_data`` will be available as data inside :ref:`filter_string` and can be very powerful
-together with internal needs information to filter needs.
+.. code-block:: rst
 
-The defined extra filter data can be used like this:
+   .. needtable::
+      :filter: var.cpu == "arm" and var.build.debug == True
 
-.. code-block:: rst
+   .. needlist::
+      :filter: "feature_a" in var.build.features
+
+**Allowed data shape:**
+
+- Top-level must be a dictionary with string keys.
+- Intermediate nodes: dictionaries only.
+- Leaf values: ``str``, ``bool``, ``int``, ``float``, or uniform lists of these types.
+- Arrays must be uniform (all elements the same scalar type).
+- No ``None`` values, no dictionaries inside arrays.
+
+Accessing a missing key raises an ``AttributeError``, which helps catch typos in filter expressions.
+
+Default: ``{}``
+
+.. seealso::
+
+   :ref:`needs_variant_data_file` for loading variant data from a JSON file.
+
+.. _`needs_variant_data_file`:
+
+needs_variant_data_file
+~~~~~~~~~~~~~~~~~~~~~~~
+
+.. versionadded:: 8.2.0
+
+Path to a JSON file containing variant data. The file is loaded and its contents
+are made available under the ``var`` namespace, just like :ref:`needs_variant_data`.
+
+If both ``needs_variant_data_file`` and ``needs_variant_data`` are set, the file is loaded first
+and the inline dictionary is deep-merged on top (inline values win on conflict).
 
-   .. needextend:: type == "req" and sphinx_tag in tags
-      :+tags: my_external_tag
+The path is resolved relative to the Sphinx ``confdir`` (the directory containing ``conf.py``).
 
-or if project has :ref:`needs_fields` defined like:
+Configuration example:
 
 .. code-block:: python
 
-   needs_fields = {'variant': {}}
+   needs_variant_data_file = "variants/arm_debug.json"
 
-The defined extra filter data can also be used like:
+Where ``variants/arm_debug.json`` contains:
 
-.. code-block:: rst
+.. code-block:: json
 
-   .. needlist::
-      :filter: variant != current_variant
+   {
+     "cpu": "arm",
+     "debug": true,
+     "build": {
+       "optimization": 2,
+       "features": ["feature_a", "feature_b"]
+     }
+   }
+
+This is particularly useful for switching configurations at build time:
 
-   .. needextract::
-      :filter: type == "story" and variant == current_variant
-      :layout: clean
-      :style: green_border
+.. code-block:: bash
+
+   sphinx-build -D needs_variant_data_file=variants/x86_release.json docs/ _build/
+
+Default: ``None``
 
 .. _`needs_allow_unsafe_filters`:
 
@@ -2472,6 +2518,37 @@ final need and schema looks like.
 Deprecated Options
 ------------------
 
+.. _`needs_filter_data`:
+
+needs_filter_data
+~~~~~~~~~~~~~~~~~
+
+.. deprecated:: 8.2.0
+
+   Use :ref:`needs_variant_data` and :ref:`needs_variant_data_file` instead.
+   These provide a dedicated ``var`` namespace that avoids collisions with need field names
+   and supports nested structured data.
+
+This option allows to use custom data inside a :ref:`filter_string`.
+
+Configuration example:
+
+.. code-block:: python
+
+   def custom_defined_func():
+       return "my_tag"
+
+   needs_filter_data = {
+       "current_variant": "project_x",
+       "sphinx_tag": custom_defined_func(),
+   }
+
+The defined ``needs_filter_data`` must be a dictionary. Its values can be a string variable or a custom defined
+function. The function gets executed during config loading and must return a string.
+
+The value of ``needs_filter_data`` will be available as data inside :ref:`filter_string` and can be very powerful
+together with internal needs information to filter needs.
+
 .. _`needs_extra_options`:
 
 needs_extra_options
@@ -2672,7 +2749,7 @@ A match expression is a string, using Python syntax, that will be evaluated agai
 - ``is_import`` (``bool``)
 - :ref:`needs_fields`
 - :ref:`needs_links` (``tuple[str, ...]``)
-- :ref:`needs_filter_data`
+- :ref:`needs_variant_data` (via the ``var`` namespace)
 
 If no predicates match, the ``default`` value is used (if present).
 

docs/directives/need.rst

@@ -319,7 +319,7 @@ The option activates jinja-parsing for the content of a need.
 If the value is set to ``true``, you can specify `Jinja <https://jinja.palletsprojects.com/>`_ syntax in the content.
 
 The **:jinja_content:** option give access to all need data, including the original content
-and the data in :ref:`needs_filter_data`.
+and the data in :ref:`needs_variant_data`.
 
 If you set the option to **False**, you deactivate jinja-parsing for the need's content.
 

docs/dynamic_functions.rst

@@ -170,7 +170,7 @@ Rules for specifying variant definitions
 * Variants gets checked from left to right.
 * When evaluating a variant definition, we use data from the current need object,
   `Sphinx-Tags <https://www.sphinx-doc.org/en/master/man/sphinx-build.html#cmdoption-sphinx-build-t>`_,
-  and :ref:`needs_filter_data` as the context for filtering.
+  and :ref:`needs_variant_data` as the context for filtering.
   Sphinx tags are injected under the name ``build_tags`` as a set of strings.
 * You can set a *need option* to multiple variant definitions by separating each definition with either
   the ``,`` symbol, like ``var_a:open, ['name' in tags]:assigned``.|br|

docs/filter.rst

@@ -189,6 +189,45 @@ If it is invalid or returns False, the related need is not taken into account fo
       .. needlist::
          :filter: "filter_example" in tags and (("B" in tags or ("spec" == type and "closed" == status)) or "test" == type)
 
+.. _filter_variant_data:
+
+Using variant data (``var``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. versionadded:: 8.2.0
+
+If :ref:`needs_variant_data` or :ref:`needs_variant_data_file` is configured,
+a ``var`` namespace is available in filter expressions, providing dot-access to nested structured data.
+
+This is useful for switching build outputs based on external configuration (CPU architecture, debug/release mode, feature flags, etc.)
+without polluting the need field namespace.
+
+The dot-access syntax mirrors `Jinja2 template syntax <https://jinja.palletsprojects.com/en/latest/templates/#variables>`__,
+making it familiar to Sphinx users.
+
+**Examples:**
+
+.. code-block:: rst
+
+   .. needtable::
+      :filter: var.cpu == "arm"
+
+   .. needlist::
+      :filter: var.build.debug == True and var.build.optimization > 1
+
+   .. needtable::
+      :filter: "feature_a" in var.build.features
+
+Accessing a key that doesn't exist raises an error, which helps catch typos:
+
+.. code-block:: rst
+
+   .. needlist::
+      :filter: var.typo == "x"
+      .. this will raise AttributeError: Unknown variant key: var.typo
+
+See :ref:`needs_variant_data` for configuration details and allowed data shapes.
+
 .. _filter_current_page:
 
 Filtering for needs on the current page
@@ -248,6 +287,8 @@ To improve performance, certain common patterns are identified and optimized by
 - ``status == 'value'`` / ``status == "value"`` / ``'value' == status`` / ``"value" == status``
 - ``status in ['value1', 'value2', ...]`` / ``status in ("value1", "value2", ...)``
 - ``'value' in tags`` / ``"value" in tags``
+- ``var.key == 'value'`` / ``var.nested.key == 'value'`` (see :ref:`filter_variant_data`)
+- ``'value' in var.key`` / ``'value' not in var.key``
 
 Also filters containing ``and`` will be split into multiple filters and evaluated separately for the above patterns.
 For example, ``type == 'spec' and other == 'value'`` will first be filtered performantly by ``type == 'spec'`` and then the remaining needs will be filtered by ``other == 'value'``.

sphinx_needs/config.py

@@ -732,7 +732,42 @@ def functions(self) -> Mapping[str, NeedFunctionsType]:
     filter_data: dict[str, str] = field(
         default_factory=dict, metadata={"rebuild": "html", "types": ()}
     )
-    """Additional context data for filters."""
+    """Additional context data for filters.
+
+    .. deprecated:: 8.2.0
+        Use :confval:`needs_variant_data` with the ``var.*`` namespace instead.
+    """
+    variant_data: dict[str, Any] = field(
+        default_factory=dict, metadata={"rebuild": "html", "types": (dict,)}
+    )
+    """Nested variant data accessible as ``var.*`` in filter expressions.
+
+    Supports nested dicts with scalar leaf values (str, bool, int, float)
+    and uniform-type arrays of scalars.
+
+    Example::
+
+        needs_variant_data = {
+            "cpu": "arm",
+            "debug": True,
+            "build": {"optimization": 2},
+        }
+
+    Then in filters: ``var.cpu == "arm"`` or ``var.build.optimization > 1``.
+    """
+    variant_data_file: str | None = field(
+        default=None,
+        metadata={
+            "rebuild": "html",
+            "types": (str, type(None)),
+            "toml_convert": _abs_path,
+        },
+    )
+    """Path to a JSON file containing variant data.
+
+    If both this and :confval:`needs_variant_data` are set, the file is loaded first
+    and inline values are deep-merged on top (inline wins).
+    """
     allow_unsafe_filters: bool = field(
         default=False, metadata={"rebuild": "html", "types": (bool,)}
     )

sphinx_needs/filter_common.py

@@ -541,8 +541,14 @@ def filter_needs_and_parts(
 
     # === Fast path: try compiled predicate to avoid eval() entirely ===
     simple_pred = try_build_simple_predicate(filter_string)
+    var_proxy = config.variant_data_proxy
+
     if simple_pred is not None:
-        fallback = config.filter_data or None
+        fallback: dict[str, Any] | None = None
+        if config.filter_data or var_proxy is not None:
+            fallback = dict(config.filter_data) if config.filter_data else {}
+            if var_proxy is not None:
+                fallback["var"] = var_proxy
         found_needs: list[NeedItem | NeedPartItem] = []
         error_reported = False
         for filter_need in needs:
@@ -631,12 +637,14 @@ def apply_default_predicate(
 
     :raises NeedsInvalidFilter: if the predicate is not valid
     """
-    predicate_context = {
+    predicate_context: dict[str, Any] = {
         **context,
         **extras,
         **links,
         **config.filter_data,
     }
+    if (var_proxy := config.variant_data_proxy) is not None:
+        predicate_context["var"] = var_proxy
     try:
         # Set filter_context as globals and not only locals in eval()!
         # Otherwise, the vars not be accessed in list comprehensions.
@@ -657,6 +665,8 @@ def filter_import_item(
     filter_context = context.copy()
     # Get needs external filter data and merge to filter_context
     filter_context.update(config.filter_data)
+    if (var_proxy := config.variant_data_proxy) is not None:
+        filter_context["var"] = var_proxy
     filter_context["search"] = need_search
     try:
         # Set filter_context as globals and not only locals in eval()!
@@ -694,10 +704,16 @@ def filter_single_need(
 
     :return: True, if need passes the filter_string, else False
     """
+    var_proxy = config.variant_data_proxy
+
     # === Fast path: short-circuit simple expressions ===
     simple_pred = try_build_simple_predicate(filter_string)
     if simple_pred is not None:
-        fallback = config.filter_data or None
+        fallback: dict[str, Any] | None = None
+        if config.filter_data or var_proxy is not None:
+            fallback = dict(config.filter_data) if config.filter_data else {}
+            if var_proxy is not None:
+                fallback["var"] = var_proxy
         try:
             result = simple_pred(need, fallback)
             if not isinstance(result, bool):
@@ -721,6 +737,8 @@ def filter_single_need(
 
     # Get needs external filter data and merge to filter_context
     filter_context.update(config.filter_data)
+    if var_proxy is not None:
+        filter_context["var"] = var_proxy
 
     filter_context["search"] = need_search
 

sphinx_needs/functions/functions.py

@@ -281,6 +281,7 @@ def resolve_functions(
 ) -> None:
     """Resolve all dynamic/variant functions in all needs."""
     needs_schema = SphinxNeedsData(app.env).get_schema()
+    var_proxy = needs_config.variant_data_proxy
     for need in needs.values():
         if not need.has_dynamic_fields:
             continue
@@ -317,6 +318,8 @@ def resolve_functions(
                             **needs_config.filter_data,
                             "build_tags": set(app.builder.tags),
                         }
+                        if var_proxy is not None:
+                            var_context["var"] = var_proxy
                         if (
                             var_return := _get_variant(
                                 item, needs_config.variants, var_context