feat(silo-import, helm, website): Revive hierarchical filter functionality (#6575)

This PR reintroduces hierarchical filtering functionality that was added
in #6302 and
#6404 but reverted so
that we could roll out the required breaking change from `hostTaxonId:
int` to `hostTaxonId: string` on its own first.

Below is an updated description taken from the original PR:

## Summary

Introduces a new "hierarchical filter" concept for metadata fields whose
values form a hierarchy resolved dynamically at SILO import time (rather
than statically pinned per pipeline version, the way lineageSystem
works). The first use case is hostTaxonId: at import time the SILO
importer collects the set of host taxon IDs observed in the released
data, asks the taxonomy service to render a SILO-format lineage YAML for
those taxa, and writes it into SILO's input directory. SILO then indexes
the column as a lineage, which gives the website a hierarchical search
UI ("Culicidae" matches records of all mosquito species, etc.).

This PR introduces changes to the helm chart, the SILO importer, and the
website.

## Helm / config

- New metadata-field property hierarchicalFilter: <URL>. Mutually
exclusive with lineageSystem (enforced in values.schema.json). When set,
the field is treated like a lineage system for SILO indexing
(generateLineageIndex) and surfaces a new hierarchicalSearch: true flag
in the website config.
- _hierarchical-filters-for-organism.tpl mirrors the existing
lineageSystemForOrganism helper. The SILO database config combines both
lists into lineageDefinitionFilenames.
- silo-deployment.yaml builds and injects the HIERARCHICAL_FILTERS env
var (JSON map) consumed by the importer.
- West-nile given hierarchicalFilter: *taxonomy_service_url to observe
the new feature on previews.
- New section in helm-chart-config.mdx documenting the feature.

## SILO import

- HIERARCHICAL_FILTERS env var parsed at startup. 
- update_hierarchical_filters checks whether the most recent import
cycle returned new values for the hierarchical filter field, and fetches
an updated lineage if so. The response is written to
<input_dir>/<field>.yaml. If the service responds 413 (file exceeds
SILO's Jackson YAML size limit), the importer retries with prune=true.
- New tests in tests/test_runner.py cover both the happy path and the
skip-when-unchanged path; new tests in tests/test_config.py cover
env-var parsing.

## Frontend

- New config options (types/config.ts). Two optional fields added to the
metadata schema: hierarchicalSearch: boolean — opt a field into the
hierarchical filter UI. hierarchicalSearchLabel: string — custom label
for the "include sublineages/subcategories" checkbox

- LineageField.tsx is renamed and extended to HierarchicalField.tsx. The
component was now has a mode prop ('lineage' | 'default') with a
MODE_CONFIGS table:

| | `lineage` mode | `default` (hierarchical) mode |
|---|---|---|
| default "include sublineages" | off | **on** |
| `showAlias` | false | **true** |
| checkbox label | "include sublineages" | "include subcategories" |
| `includeZeroCounts` | true | true |

- SearchForm.tsx renders the HierarchicalField when either
field.lineageSearch or field.hierarchicalSearch is set, passing
mode='lineage' for the former and 'default' for the latter.

- The lineage options hook in AutoCompleteOptions.ts gains two
parameters: 1) showAlias — when a lineage/node has aliases, display the
first alias as the user-facing label (the value stays the canonical
name). This is the hierarchical filter's user-facing label. 2)
includeZeroCounts — when false, options with a count of 0 are filtered
out.

- Added a valueToLabel map to SingleChoiceAutoCompleteField.tsx so the
input's displayValue shows the human-readable label (e.g., the alias)
instead of the raw stored value.

## Screenshots & manual testing
I configured a hierarchical filter for hostTaxonId and set up a preview.
On the preview, I tested filtering west-nile sequences using the new
hierarchical filter autocomplete. It seems to work as intended, and also
the interplay with the existing host name - common / scientific looks
reasonable: if I set a filter for the species 'Culex' using host
name-scientific, all the options with zero count (birds etc.) no longes
show up in the hierarchical filter autocomplete.

This is how the autocomplete looks:

<img width="274" height="122" alt="grafik"
src="https://github.com/user-attachments/assets/8224989c-a64e-4154-b45a-72f4551faa0b"
/>


<img width="274" height="343" alt="grafik"
src="https://github.com/user-attachments/assets/89b0eb28-4f9e-4dd8-9273-a01dd0702d65"
/>



### PR Checklist
- [ ] All necessary documentation has been adapted.
- [ ] The implemented feature is covered by appropriate, automated
tests.
- [ ] Any manual testing that has been done is documented (i.e. what
exactly was tested?)

🚀 Preview: https://revive-hierarchical-filte.loculus.org

---------

Co-authored-by: Anna (Anya) Parker <50943381+anna-parker@users.noreply.github.com>

Author: maverbiest

docs/src/content/docs/reference/helm-chart-config.mdx

@@ -81,6 +81,15 @@ lineageSystemDefinitions:
 
 Note: Lineages should be determined by the preprocessing pipeline and not the user, set the lineage field to `noInput: true` to reflect this or alternatively only allow users to submit valid lineage values by adding an `options` list.
 
+### Hierarchical filter sources
+
+For metadata fields whose values form a hierarchy that's resolved at SILO import
+time (rather than fetched as a static file per pipeline version), set
+`hierarchicalFilter: <URL>`, where URL is the URL of the service that will create the
+hierarchical file for SILO. The importer collects observed
+values for the field, sends them to that service, and writes the returned YAML
+into SILO's input directory as `<MetadataName>.yaml`.
+
 ### Organism (type)
 
 Each organism object has the following fields:

kubernetes/loculus/templates/_common-metadata.tpl

@@ -372,6 +372,12 @@ organisms:
   {{- if .lineageSystem }}
   lineageSearch: true
   {{- end }}
+  {{- if .hierarchicalFilter }}
+  hierarchicalSearch: true
+  {{- end }}
+  {{- if and .hierarchicalFilter .hierarchicalSearchLabel }}
+  hierarchicalSearchLabel: {{ .hierarchicalSearchLabel }}
+  {{- end }}
   {{- if .hideOnSequenceDetailsPage }}
   hideOnSequenceDetailsPage: {{ .hideOnSequenceDetailsPage }}
   {{- end }}

kubernetes/loculus/templates/_hierarchical-filters-for-organism.tpl

@@ -0,0 +1,11 @@
+{{- define "loculus.hierarchicalFiltersForOrganism" -}}
+{{- $organism := . -}}
+{{- $schema := $organism.schema | include "loculus.patchMetadataSchema" | fromYaml }}
+{{- $filters := dict }}
+{{- range $entry := $schema.metadata }}
+  {{- if hasKey $entry "hierarchicalFilter" }}
+    {{- $_ := set $filters $entry.name $entry.hierarchicalFilter }}
+  {{- end }}
+{{- end }}
+{{- $filters | toYaml -}}
+{{- end }}

kubernetes/loculus/templates/_siloDatabaseConfig.tpl

@@ -1,12 +1,13 @@
 {{- define "loculus.siloDatabaseShared" -}}
 {{- $type := default "string" .type -}}
+{{- $lineageName := ternary .name (default "" .lineageSystem) (not (empty .hierarchicalFilter)) -}}
 - type: {{ ($type | eq "timestamp") | ternary "int" (($type | eq "authors") | ternary "string" $type) }}
-  {{- if .generateIndex }}
+  {{- if and .generateIndex (not $lineageName) }}
   generateIndex: {{ .generateIndex }}
   {{- end }}
-  {{- if .lineageSystem }}
+  {{- if $lineageName }}
   generateIndex: true
-  generateLineageIndex: {{ .lineageSystem }} {{- /* must match the file name in the lineageDefinitionFilenames */}}
+  generateLineageIndex: {{ $lineageName }} {{- /* must match the file name in the lineageDefinitionFilenames */}}
   {{- end }}
 {{- end }}
 

kubernetes/loculus/templates/lapis-silo-database-config.yaml

@@ -5,6 +5,11 @@
 {{- $organismContent := $item.contents }}
 
 {{- $lineageSystem := $organismContent | include "loculus.lineageSystemForOrganism" | fromYamlArray }}
+{{- $hierarchicalFilters := $organismContent | include "loculus.hierarchicalFiltersForOrganism" | fromYaml }}
+{{- $allLineageNames := $lineageSystem }}
+{{- range $name, $_ := $hierarchicalFilters }}
+  {{- $allLineageNames = append $allLineageNames $name }}
+{{- end }}
 ---
 apiVersion: v1
 kind: ConfigMap
@@ -24,10 +29,10 @@ data:
     outputDirectory: /preprocessing/output
     ndjsonInputFilename: data.ndjson.zst
     referenceGenomeFilename: reference_genomes.json
-    {{- if $lineageSystem }}
+    {{- if $allLineageNames }}
     lineageDefinitionFilenames:
-    {{- range $ls := $lineageSystem }}
-      - {{ printf "%s.yaml" $ls }}
+    {{- range $name := $allLineageNames }}
+      - {{ printf "%s.yaml" $name }}
     {{- end }}
     {{- end }}
 

kubernetes/loculus/templates/silo-deployment.yaml

@@ -5,6 +5,7 @@
 {{- $key := $item.key }}
 {{- $organismContent := $item.contents }}
 {{- $lineageSystem := $organismContent | include "loculus.lineageSystemForOrganism" | fromYamlArray }}
+{{- $hierarchicalFilters := $organismContent | include "loculus.hierarchicalFiltersForOrganism" | fromYaml }}
 ---
 apiVersion: apps/v1
 kind: Deployment
@@ -94,6 +95,10 @@ spec:
             - name: LINEAGE_DEFINITIONS
               value: {{ $defsBySystem | toJson | quote }}
             {{- end }}
+            {{- if $hierarchicalFilters }}
+            - name: HIERARCHICAL_FILTERS
+              value: {{ $hierarchicalFilters | toJson | quote }}
+            {{- end }}
             - name: SILO_RUN_TIMEOUT_SECONDS
               value: {{ $.Values.siloImport.siloTimeoutSeconds | quote }}
             - name: HARD_REFRESH_INTERVAL

kubernetes/loculus/values.schema.json

@@ -232,6 +232,16 @@
           "groups": ["metadata"],
           "type": "string",
           "description": "Use this on string fields that contain lineages, if you want to enable searches that can include sublineages. The value needs to be a lineage system that is defined under the `lineageSystemDefinitions` key."
+        },
+        "hierarchicalFilter": {
+          "groups": ["metadata"],
+          "type": "string",
+          "description": "Use this on string fields whose values form a hierarchy resolved at SILO import time. The value should be an URL to the lineage file."
+        },
+        "hierarchicalSearchLabel": {
+          "groups": ["metadata"],
+          "type": "string",
+          "description": "CheckBox label for hierarchical search - only applies when using hierarchicalFilter."
         }
       },
       "required": ["name"],
@@ -290,6 +300,24 @@
             "errorMessage": "When lineageSystem is set, type must be 'string' or 'authors'"
           }
         },
+        {
+          "if": {
+            "properties": { "hierarchicalFilter": { "type": "string" } },
+            "required": ["hierarchicalFilter"]
+          },
+          "then": {
+            "properties": {
+              "type": { "enum": ["string", "authors"] }
+            },
+            "errorMessage": "When hierarchicalFilter is set, type must be 'string' or 'authors'"
+          }
+        },
+        {
+          "not": {
+            "description": "lineageSystem and hierarchicalFilter cannot both be set",
+            "required": ["lineageSystem", "hierarchicalFilter"]
+          }
+        },
         {
           "if": {
             "properties": { "rangeSearch": { "const": true } },

kubernetes/loculus/values.yaml

@@ -1256,7 +1256,7 @@ defaultOrganismConfig: &defaultOrganismConfig
             hostNameScientific: hostNameScientific
             hostTaxonId: hostTaxonId
           args:
-            taxonomy_service_url: "http://loculus-taxonomy-service:5000"
+            taxonomy_service_url: &taxonomy_service_url http://loculus-taxonomy-service:5000
       - name: hostNameScientific
         displayName: Host name - scientific
         definition: "The scientific name of the host from which the sample was collected."
@@ -1276,7 +1276,7 @@ defaultOrganismConfig: &defaultOrganismConfig
             hostTaxonId: processed.hostTaxonId
             hostNameScientific: hostNameScientific
           args:
-            taxonomy_service_url: "http://loculus-taxonomy-service:5000"
+            taxonomy_service_url: *taxonomy_service_url
       - name: hostNameCommon
         displayName: Host name - common
         definition: "The common name of the host from which the sample was collected."
@@ -1295,7 +1295,7 @@ defaultOrganismConfig: &defaultOrganismConfig
           inputs:
             hostTaxonId: processed.hostTaxonId
           args:
-            taxonomy_service_url: "http://loculus-taxonomy-service:5000"
+            taxonomy_service_url: *taxonomy_service_url
       - name: isLabHost
         displayName: Is lab host
         definition: "If a laboratory host (e.g. cultured cell line) was used to propagate the sample."
@@ -1535,7 +1535,7 @@ defaultOrganismConfig: &defaultOrganismConfig
         - "prepro"
       replicas: 1
       taxonomyServiceArgs: &preprocessingTaxonomyServiceArgs
-        taxonomy_service_url: "http://loculus-taxonomy-service:5000"
+        taxonomy_service_url: *taxonomy_service_url
       configFile: &preprocessingConfigFile
         log_level: DEBUG
         batch_size: 100
@@ -1657,6 +1657,9 @@ defaultOrganisms:
               regex_pattern: '^(?:[^/]+/)?[^/]+/(?P<identifier>[^/]+)/\d{4}(?:-\d{2}){0,2}$'
         - <<: *hostTaxonIdConfig
           required: true
+          hierarchicalFilter: *taxonomy_service_url
+          hierarchicalSearchLabel: "include subtaxa"
+          initiallyVisible: true
         - name: lineage
           displayName: Lineage
           definition: "Assigned clade label from the nearest reference tree node."

loculus-silo/README.md

@@ -4,9 +4,10 @@ Python service that retrieves released datasets from the Loculus backend, transf
 for SILO, and runs SILO preprocessing directly.
 
 The importer downloads data from the Loculus backend, [transforms it into the format required by SILO](https://github.com/GenSpectrum/LAPIS-SILO/tree/main/tools/legacyNdjsonTransformer), and runs SILO preprocessing only if the following conditions hold:
+
 1. The data is in a valid format (e.g. ndjson format where each line is a valid json, has number of expected records, and the pipeline version exists and is a valid integer if a lineage definition is required).
 2. The lineage definitions file can be produced if it is required.
-3. The data has changed since the last download or it has been over more than `HARD_REFRESH_INTERVAL` since the last hard refresh. We determine if the data has changed from the header (e.g. 304 not modified) and by comparing a hash of the data. 
+3. The data has changed since the last download or it has been over more than `HARD_REFRESH_INTERVAL` since the last hard refresh. We determine if the data has changed from the header (e.g. 304 not modified) and by comparing a hash of the data.
 
 ## Local development
 

loculus-silo/src/silo_import/config.py

@@ -5,6 +5,9 @@
 from dataclasses import dataclass
 from pathlib import Path
 
+MetadataField = str
+HierarchicalServiceUrl = str
+
 
 @dataclass(frozen=True)
 class ImporterConfig:
@@ -16,6 +19,7 @@ class ImporterConfig:
     root_dir: Path
     silo_binary: Path
     preprocessing_config: Path
+    hierarchical_filters: dict[MetadataField, HierarchicalServiceUrl] | None = None
 
     @classmethod
     def from_env(cls) -> ImporterConfig:
@@ -47,6 +51,8 @@ def from_env(cls) -> ImporterConfig:
             except TypeError as exc:
                 raise RuntimeError(str(exc)) from exc
 
+        hierarchical_filters = _parse_hierarchical_filters(env.get("HIERARCHICAL_FILTERS"))
+
         hard_refresh_interval = int(env.get("HARD_REFRESH_INTERVAL", "3600"))
         poll_interval = int(env.get("SILO_IMPORT_POLL_INTERVAL_SECONDS", "30"))
         silo_run_timeout = int(env.get("SILO_RUN_TIMEOUT_SECONDS", "3600"))
@@ -66,8 +72,28 @@ def from_env(cls) -> ImporterConfig:
             root_dir=root_dir,
             silo_binary=silo_binary,
             preprocessing_config=preprocessing_config,
+            hierarchical_filters=hierarchical_filters,
         )
 
     @property
     def released_data_endpoint(self) -> str:
         return f"{self.backend_base_url}/get-released-data?compression=zstd"
+
+
+def _parse_hierarchical_filters(
+    raw: str | None,
+) -> dict[MetadataField, HierarchicalServiceUrl] | None:
+    if not raw:
+        return None
+
+    try:
+        data = json.loads(raw)
+    except json.JSONDecodeError as exc:
+        msg = "HIERARCHICAL_FILTERS must be valid JSON"
+        raise RuntimeError(msg) from exc
+
+    if not isinstance(data, dict):
+        msg = f"HIERARCHICAL_FILTERS must be a JSON object, got: {raw}"
+        raise TypeError(msg)
+
+    return data or None