Enhance regex support in search criteria and documentation updates for clarity

Author: dwr-psandhu

README-mathref.md

@@ -179,7 +179,7 @@ Each expression variable is a short alias resolved by catalog attribute criteria
       variable: precipitation
       interval: hourly
 
-# Multi-station mean: _require_single: false concatenates ALL matches into a DataFrame
+# Multi-station mean: match_all: true concatenates ALL matches into a DataFrame
 - name: mean_wind_speed__all_stations__hourly
   expression: ws.mean(axis=1)
   variable: mean_wind_speed
@@ -189,10 +189,109 @@ Each expression variable is a short alias resolved by catalog attribute criteria
     ws:
       variable: wind_speed
       interval: hourly
-      _require_single: false
+      match_all: true
 ```
 
-> **`_require_single: false`** — when set inside a `search_map` criteria block, *all* catalog entries matching those criteria are fetched and concatenated column-wise (axis=1 join by timestamp index) so the alias resolves to a DataFrame. Use this for multi-station aggregates such as `ws.mean(axis=1)`.
+> **`match_all: true`** — when set inside a `search_map` criteria block, *all* catalog entries matching those criteria are fetched and concatenated column-wise (axis=1 join by timestamp index) so the alias resolves to a DataFrame. Use this for multi-station aggregates such as `ws.mean(axis=1)`.
+
+### Regex matching in criteria
+
+By default, every `attr=value` criterion in a `search_map` block (or in a `catalog.search()` call) performs an **exact equality check**. You can switch to a **case-insensitive regular expression** by using `~` instead of `=` as the operator, or by passing a tilde-prefixed string in code.
+
+#### Rules
+
+| Rule | Detail |
+|---|---|
+| Operator | `~` in the editor/YAML; `"~pattern"` (tilde-prefixed string) in Python |
+| Matching | `re.fullmatch` — the pattern must match the **entire** attribute value |
+| Case | Case-insensitive (`re.IGNORECASE`) |
+| Partial match | Use `.*` to allow leading/trailing characters (e.g. `~EC.*`) |
+
+#### YAML examples
+
+```yaml
+search_map:
+  # Exact match (default) — only "wind_speed" qualifies, not "wind_speed_mph"
+  obs:
+    variable: wind_speed
+    interval: hourly
+
+  # Regex fullmatch — matches "wind_speed" AND "wind_speed_mph" (starts with "wind_speed")
+  obs_any:
+    variable: ~wind_speed.*
+    interval: hourly
+
+  # Case-insensitive — "EC", "ec", "Ec" all match
+  salinity:
+    variable: ~ec
+    interval: ~15min.*
+
+  # Match several known prefixes with alternation
+  flow_or_stage:
+    variable: ~(flow|stage)
+    interval: hourly
+```
+
+#### Python `set_search()` / `search_map` dict examples
+
+```python
+# Regex via set_search() — matches "EC", "ec_hourly", "EC_daily", etc.
+m = (MathDataReference("obs * 1.0", name="ec_copy")
+     .set_search("obs", variable="~EC.*", interval="hourly")
+     .set_catalog(catalog))
+
+# Regex in a search_map dict passed to the constructor
+m = MathDataReference(
+    "a + b",
+    search_map={
+        "a": {"variable": "~flow.*", "location": "upstream"},
+        "b": {"variable": "~flow.*", "location": "downstream"},
+    },
+    name="net_flow",
+)
+
+# catalog.search() with regex — find all refs whose name starts with "station"
+refs = catalog.search(name="~station.*")
+# catalog.search() with regex attribute — find all EC-family variables
+refs = catalog.search(variable="~EC.*")
+```
+
+#### Using regex in the editor
+
+Type criteria using `~` instead of `=` to indicate regex:
+
+```
+variable~EC.*
+variable~EC.*, interval=hourly
+```
+
+The **`+attr`** picker always appends `attr=`. To switch to regex, manually replace `=` with `~` in the criteria field after picking the attribute name.
+
+#### Partial vs full match
+
+`re.fullmatch` is used, so the pattern must match the **entire** attribute value, not just a substring:
+
+| Pattern | Attribute value | Matches? |
+|---|---|---|
+| `~EC` | `"EC"` | ✅ yes — exact fullmatch |
+| `~EC` | `"EC_daily"` | ❌ no — `EC` alone doesn't cover `_daily` |
+| `~EC.*` | `"EC_daily"` | ✅ yes — `.*` absorbs the suffix |
+| `~ec` | `"EC"` | ✅ yes — case-insensitive |
+| `~ec` | `"EC_DAILY"` | ❌ no — same fullmatch rule applies |
+| `~ec.*` | `"EC_DAILY"` | ✅ yes |
+| `~(flow\|stage)` | `"flow"` | ✅ yes |
+| `~(flow\|stage)` | `"velocity"` | ❌ no |
+
+#### Limitation — literal `~` values
+
+If a metadata attribute value legitimately starts with `~` (unusual), it **cannot** be matched using `=` with a tilde-prefixed expected value, because any value beginning with `~` is interpreted as a regex pattern. As a workaround, use a callable predicate in Python:
+
+```python
+# Match the literal value "~special" exactly
+refs = catalog.search(variable=lambda v: v == "~special")
+```
+
+This limitation does not apply to the YAML or editor format, where you would simply use `variable=~special` to trigger regex mode.
 
 ### Chaining Math References
 
@@ -278,8 +377,8 @@ One row per expression variable. Rows can be added with **+ Add variable** and r
 | Field | Purpose |
 |---|---|
 | **Alias** | Short identifier used in the Expression (e.g. `obs`) |
-| **Match all** | When checked, all matching catalog entries are concatenated into a DataFrame (equivalent to `_require_single: false`) |
-| **Catalog criteria** | Comma-separated `attr=val` pairs used to search the catalog (e.g. `variable=wind_speed, interval=hourly`) |
+| **Match all** | When checked, all matching catalog entries are concatenated into a DataFrame (equivalent to `match_all: true` in YAML) |
+| **Catalog criteria** | Comma-separated `attr=val` (exact) or `attr~regex` (regex fullmatch) pairs used to search the catalog (e.g. `variable=wind_speed, interval=hourly` or `variable~EC.*, interval=hourly`) |
 | **+ attr** | Drop-down picker; selecting an attribute name appends `attr=` to the criteria field so you only need to fill in the value |
 | **▶** | Per-row test button — runs the criteria against the live catalog and shows an inline badge: `✅ 1 match`, `⚠️ N matches`, or `❌ 0 matches`. If exactly one match is found and the **Attributes** field is empty, it is auto-filled with the match's identifying attributes |
 

dvue/catalog.py

@@ -296,6 +296,33 @@ def _resolve_class(fqcn: str) -> type:
     return getattr(module, class_name)
 
 
+def _criterion_matches(actual: Any, expected: Any) -> bool:
+    """Return ``True`` if *actual* satisfies the *expected* criterion.
+
+    * ``expected`` is a **string starting with "~"** — treated as a
+      case-insensitive regular expression; ``re.fullmatch`` is used so the
+      pattern must cover the entire attribute value.  Use ``~EC.*`` (not
+      ``~EC``) to match values that *start with* ``EC``.
+    * ``expected`` is a **callable** — delegated to the caller (not handled
+      here; callers should check ``callable(expected)`` first).
+    * Otherwise — exact equality with transparent type coercion when scalar
+      types differ (e.g. string ``'0'`` vs integer ``0``).
+    """
+    if isinstance(expected, str) and expected.startswith("~"):
+        pattern = expected[1:]
+        return bool(re.fullmatch(pattern, str(actual) if actual is not None else "", re.IGNORECASE))
+    if actual != expected:
+        # Try type coercion when scalar types differ.
+        try:
+            if type(actual) is not type(expected):
+                if actual == type(actual)(expected):
+                    return True
+        except (ValueError, TypeError):
+            pass
+        return False
+    return True
+
+
 # ---------------------------------------------------------------------------
 # DataReference
 # ---------------------------------------------------------------------------
@@ -519,6 +546,8 @@ def matches(self, **criteria: Any) -> bool:
         Each criterion value can be:
 
         * a **scalar** – exact equality check.
+        * a **string starting with "~"** – case-insensitive regex fullmatch
+          (e.g. ``variable="~EC.*"`` matches ``"EC_daily"`` or ``"ec_hourly"``).
         * a **callable** ``f(value) -> bool`` – custom predicate.
         """
         for key, expected in criteria.items():
@@ -530,14 +559,7 @@ def matches(self, **criteria: Any) -> bool:
             if callable(expected):
                 if not expected(actual):
                     return False
-            elif actual != expected:
-                # Try type coercion when scalar types differ 
-                try:
-                    if type(actual) is not type(expected):
-                        if actual == type(actual)(expected):
-                            continue
-                except (ValueError, TypeError):
-                    pass
+            elif not _criterion_matches(actual, expected):
                 return False
         return True
 
@@ -1078,12 +1100,17 @@ def search(self, **criteria: Any) -> List[DataReference]:
         Each criterion value may be:
 
         * a **scalar** – exact equality check.
+        * a **string starting with "~"** – case-insensitive regex fullmatch
+          (e.g. ``name="~station.*"`` matches any name beginning with
+          ``station``, case-insensitively).
         * a **callable** ``f(value) -> bool`` – custom predicate.
 
         Examples
         --------
         >>> catalog.search(variable="temperature")
         >>> catalog.search(name="my_ref")
+        >>> catalog.search(name="~station.*")           # regex on name
+        >>> catalog.search(variable="~EC.*")            # regex on attribute
         >>> catalog.search(year=lambda y: int(y) >= 2020)
         >>> catalog.search(variable="temperature", unit="degC")
         """
@@ -1097,7 +1124,7 @@ def search(self, **criteria: Any) -> List[DataReference]:
                 if callable(name_criterion):
                     if not name_criterion(r.name):
                         continue
-                elif r.name != name_criterion:
+                elif not _criterion_matches(r.name, name_criterion):
                     continue
             if r.matches(**raw_criteria):
                 results.append(r)

dvue/math_ref_editor.py

@@ -112,7 +112,13 @@ def _parse_attrs(text: str) -> Dict[str, str]:
 
     @staticmethod
     def _parse_search_map(text: str):
-        """Parse ``var[multi]: key=val, key=val`` lines.
+        """Parse ``var[multi]: key=val, key~regex, …`` lines.
+
+        Supports two operators per attribute token:
+
+        * ``attr=value`` — exact match (stored as the plain string ``value``).
+        * ``attr~pattern`` — regex fullmatch, case-insensitive (stored as
+          ``"~pattern"`` so the tilde prefix is visible in round-trips).
 
         Returns
         -------
@@ -135,7 +141,15 @@ def _parse_search_map(text: str):
             criteria: Dict[str, str] = {}
             for part in criteria_str.split(","):
                 part = part.strip()
-                if "=" in part:
+                # Detect operator: ~ (regex) takes priority over = (exact).
+                tilde_pos = part.find("~")
+                eq_pos = part.find("=")
+                if tilde_pos != -1 and (eq_pos == -1 or tilde_pos < eq_pos):
+                    k, _, v = part.partition("~")
+                    k = k.strip()
+                    if k:
+                        criteria[k] = "~" + v.strip()
+                elif eq_pos != -1:
                     k, _, v = part.partition("=")
                     criteria[k.strip()] = v.strip()
             if var and criteria:
@@ -145,12 +159,22 @@ def _parse_search_map(text: str):
 
     @staticmethod
     def _render_search_map(search_map: Dict[str, Any], req: Dict[str, bool]) -> str:
-        """Render ``search_map`` + ``search_require_single`` to editor text."""
+        """Render ``search_map`` + ``search_require_single`` to editor text.
+
+        Regex criteria (stored with a ``~`` prefix) are emitted as
+        ``attr~pattern``; exact criteria are emitted as ``attr=value``.
+        """
         lines = []
         for var, criteria in search_map.items():
             require_single = req.get(var, True)
             tag = "" if require_single else "[multi]"
-            criteria_str = ", ".join(f"{k}={v}" for k, v in criteria.items())
+            parts = []
+            for k, v in criteria.items():
+                if isinstance(v, str) and v.startswith("~"):
+                    parts.append(f"{k}{v}")  # e.g. "variable~EC.*"
+                else:
+                    parts.append(f"{k}={v}")
+            criteria_str = ", ".join(parts)
             lines.append(f"{var}{tag}: {criteria_str}")
         return "\n".join(lines)
 
@@ -306,7 +330,7 @@ def _on_insert(ev: Any) -> None:
             pn.pane.Markdown("**Alias**", width=100, margin=(0, 4, 0, 4)),
             pn.pane.Markdown("**Match all**", width=80, margin=(0, 4, 0, 4)),
             pn.pane.Markdown(
-                "**Catalog criteria** (`attr=val, attr=val …`)",
+                "**Catalog criteria** (`attr=val` exact · `attr~regex` pattern)",
                 sizing_mode="stretch_width",
                 margin=(0, 4, 0, 4),
             ),
@@ -426,11 +450,19 @@ def _on_row_test(ev: Any) -> None:
                 criteria: Dict[str, str] = {}
                 for part in crit_text.split(","):
                     part = part.strip()
-                    if "=" in part:
+                    # Detect operator: ~ (regex) takes priority over = (exact).
+                    tilde_pos = part.find("~")
+                    eq_pos = part.find("=")
+                    if tilde_pos != -1 and (eq_pos == -1 or tilde_pos < eq_pos):
+                        k, _, v = part.partition("~")
+                        k = k.strip()
+                        if k:
+                            criteria[k] = "~" + v.strip()
+                    elif eq_pos != -1:
                         k, _, v = part.partition("=")
                         criteria[k.strip()] = v.strip()
                 if not criteria:
-                    _row_result_md.object = "⚠️ No valid `attr=val` pairs found."
+                    _row_result_md.object = "⚠️ No valid `attr=val` or `attr~regex` pairs found."
                     return
                 dataui.set_progress(-1)
                 try:

examples/data/math_refs.yaml

@@ -59,7 +59,7 @@
 
 # -- 4. Multi-station mean wind speed (all matching stations) -----------------
 #
-# _require_single: false tells the resolver to concat all matching references
+# match_all: true tells the resolver to concat all matching references
 # into a DataFrame.  ws.mean(axis=1) then computes the column-wise mean so
 # the result is a single Series regardless of how many stations match.
 - name: mean_wind_speed__all_stations__hourly
@@ -71,4 +71,4 @@
     ws:
       variable: wind_speed
       interval: hourly
-      _require_single: false
+      match_all: true

examples/data/math_refs_search_map.yaml

@@ -76,7 +76,7 @@
       variable: precipitation
       interval: hourly
 
-# -- Example 4: Multi-station mean with _require_single: false ----------------
+# -- Example 4: Multi-station mean with match_all: true ----------------------
 # ws resolves to ALL catalog entries where variable=wind_speed + interval=hourly
 # (one per station).  They are concatenated into a DataFrame (axis=1 join by
 # timestamp index) so ws.mean(axis=1) produces the mean across all stations.
@@ -93,7 +93,7 @@
     ws:
       variable: wind_speed
       interval: hourly
-      _require_single: false
+      match_all: true
 
 # -- Example 5: Chain via catalog name (MathDataReference → MathDataReference) --
 # References wind_speed_mph__A__hourly, itself a MathDataReference defined