Skip to content

Commit f1588ae

Browse files
committed
docs(waterdata): Frame _NEVER_CHUNK as exceptions to a default-chunk rule
After investigating: the OGC getters expose ~94 list-shaped params, all chunkable. The current 13-entry denylist captures every exception. An allowlist would be ~7x longer and would need updating every time USGS adds a column. Reframe the comment to state the default rule first ("any list-shaped kwarg gets chunked"), then enumerate the exceptions by reason (response-shaping, structured, intervals, handled-elsewhere, scalar-by- contract). Reads as "here are the few cases the default-chunk rule doesn't apply" rather than "here is an arbitrary exclusion set."
1 parent ee550be commit f1588ae

1 file changed

Lines changed: 12 additions & 7 deletions

File tree

dataretrieval/waterdata/chunking.py

Lines changed: 12 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -51,16 +51,21 @@
5151
_split_top_level_or,
5252
)
5353

54-
# Params the multi-value chunker must NOT comma-join across sub-requests.
55-
# Categorized to make additions intentional:
56-
# - response shape: ``properties`` defines columns; sharding would
57-
# give different columns per chunk.
54+
# Default rule: any list-shaped kwarg with >1 element is chunked across
55+
# sub-requests — each chunk becomes a comma-joined sub-list in the URL.
56+
# The OGC getters expose ~90 such list-shaped params (IDs, codes,
57+
# statuses, ...), all chunkable, so it's shorter to enumerate the
58+
# exceptions than to maintain an allowlist that grows with the API.
59+
# Exceptions, by reason:
60+
# - response shape: ``properties`` defines the columns; sharding
61+
# would yield different schemas per chunk.
5862
# - structured: ``bbox`` is a fixed 4-element coord tuple.
5963
# - intervals: date/time ranges are not enumerable sets.
6064
# - handled elsewhere: ``filter`` gets OR-clause partitioning in
61-
# ``_plan_joint``; multi-value semantics would
62-
# emit malformed CQL.
63-
# - scalar by contract: ``limit``, ``skip_geometry``, ``filter_lang``.
65+
# ``_plan_joint``; comma-joining CQL clauses
66+
# would emit malformed expressions.
67+
# - scalar by contract: ``limit``, ``skip_geometry``, ``filter_lang``
68+
# — a list value would be a type-erasure smuggle.
6469
_NEVER_CHUNK = frozenset(
6570
{
6671
"properties",

0 commit comments

Comments
 (0)