Budget against max per-clause encoding ratio

The whole-filter ratio is an average; a chunk that happens to contain
only the heavier-encoding clauses (e.g. heavy clauses clustered at one
end of the filter) can exceed the average ratio and push the full URL a
few bytes past _WATERDATA_URL_BYTE_LIMIT. The overflow was invisible in
practice — the 8,000 declared budget vs 8,200 observed 414 cliff gave
enough headroom — but the computed budget was technically being
violated, and a more adversarial clause mix could grow the overflow.

Compute the encoding ratio from the heaviest-encoding clause instead of
the whole filter. Adds one extra chunk on adversarial inputs (8 instead
of 7 for 100 heavy + 400 light) in exchange for every chunk provably
staying under the declared URL limit.

Verified live: the adversarial clustered-heavy filter now produces 8
chunks with max URL 7806 bytes, all returning 200 OK.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: thodson-usgs

dataretrieval/waterdata/utils.py

@@ -350,14 +350,17 @@ def _effective_filter_budget(args: dict[str, Any], filter_expr: str) -> int:
        the request with a 1-byte placeholder filter.
     2. Subtract from the URL limit to get the bytes available for the
        encoded filter value.
-    3. Convert back to raw CQL bytes using the filter's own URL-encoding
-       ratio (e.g. uniform time-interval clauses inflate ~1.4x; heavy
-       special-char clauses can inflate more).
+    3. Convert back to raw CQL bytes using the *maximum* per-clause
+       encoding ratio, not the whole-filter average. A chunk can end up
+       containing only the heavier-encoding clauses (e.g. heavy ones
+       clustered at one end of the filter), so budgeting against the
+       average lets such a chunk overflow the URL limit by a few bytes.
     """
     probe = _construct_api_requests(**{**args, "filter": "x"})
     non_filter_url_bytes = len(probe.url) - 1
     available_url_bytes = _WATERDATA_URL_BYTE_LIMIT - non_filter_url_bytes
-    encoding_ratio = len(quote_plus(filter_expr)) / len(filter_expr)
+    parts = _split_top_level_or(filter_expr) or [filter_expr]
+    encoding_ratio = max(len(quote_plus(p)) / len(p) for p in parts if p)
     return max(100, int(available_url_bytes / encoding_ratio))
 
 

tests/waterdata_utils_test.py

@@ -337,6 +337,45 @@ def test_effective_filter_budget_respects_url_limit():
     assert len(quote_plus(padded)) <= _WATERDATA_URL_BYTE_LIMIT
 
 
+def test_effective_filter_budget_uses_max_clause_ratio():
+    """Heavy clauses clustered in one part of the filter must not be able
+    to push any chunk over the URL limit. The budget is computed against
+    the max per-clause encoding ratio, not the whole-filter average, so
+    a chunk of only-heaviest-clauses still fits."""
+    from urllib.parse import quote_plus
+
+    heavy = (
+        "(time >= '2023-01-15T00:00:00Z' AND time <= '2023-01-15T00:30:00Z' "
+        "AND approval_status IN ('Approved','Provisional','Revised'))"
+    )
+    light = "(time >= '2023-01-15T00:00:00Z' AND time <= '2023-01-15T00:30:00Z')"
+    # Heavy ratio < light ratio for these shapes; cluster them at opposite
+    # ends so the chunker must produce at least one light-only chunk.
+    clauses = [heavy] * 100 + [light] * 400
+    expr = " OR ".join(clauses)
+    args = {
+        "service": "continuous",
+        "monitoring_location_id": "USGS-02238500",
+        "filter": expr,
+        "filter_lang": "cql-text",
+    }
+    budget = _effective_filter_budget(args, expr)
+    chunks = _chunk_cql_or(expr, max_len=budget)
+    assert len(chunks) > 1
+
+    # Every chunk, once built into a full request, fits under the URL byte
+    # limit — even the all-light chunks that have a higher-than-average ratio.
+    for chunk in chunks:
+        req = _construct_api_requests(**{**args, "filter": chunk})
+        assert len(req.url) <= _WATERDATA_URL_BYTE_LIMIT, (
+            f"chunk url {len(req.url)} exceeds {_WATERDATA_URL_BYTE_LIMIT}"
+        )
+
+    # Budget should be tight enough that a chunk of only-light clauses
+    # (the heavier-encoding shape here) still fits.
+    assert len(quote_plus(light)) * (budget // len(light)) < _WATERDATA_URL_BYTE_LIMIT
+
+
 def test_effective_filter_budget_shrinks_with_more_url_params():
     """Adding more scalar query params consumes URL bytes and should
     shrink the raw filter budget accordingly."""