Skip to content

Commit afcbf62

Browse files
thodson-usgsclaude
andauthored
fix(waterdata): drop id from wire properties so daily/continuous don't 400 (#323)
* fix(waterdata): drop id from the wire `properties` so daily/continuous don't 400 The USGS OGC API now rejects the feature `id` as a selectable `properties` value on the `daily` and `continuous` collections (HTTP 400 "InvalidParameterValue. unknown properties specified"; `latest-daily` still accepts it). `_switch_properties_id` translated the user-facing id column (e.g. `daily_id`) to `id` and *kept* it in the wire `properties` list, so any call passing the id column in `properties` started failing — breaking `test_get_daily_properties` and `test_get_daily_properties_id`. The feature `id` is always returned and is renamed to the service-specific id column in `_arrange_cols`, so it never needs to be requested. Drop every id alias (`id`, the service id like `daily_id`, and its singular form) plus `geometry` (controlled by `skip_geometry`) from the wire list, matching the R dataRetrieval package's `switch_properties_id`. User-facing behavior is unchanged: callers still pass `daily_id`/`id` in `properties` and get the same output columns; only the request the chunker sends is corrected. Also update the stale `test_get_continuous` geometry assertion: with `skip_geometry` unset the server now includes geometry by default, which the library already documents ("the server defaults to including geometry"). The user-facing default is intentionally left unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(waterdata): drop the dead service_id_singular branch in _switch_properties_id Both callers pass id_name = _OUTPUT_ID_BY_SERVICE[service] (the canonical service id column), so the `service.endswith("s")` singular-alias heuristic never caught a real property the drop set was missing: where it produced a meaningful name it equalled id_name already in the set, and otherwise it produced non-existent forms (e.g. "continuou_id"). The drop set is now {"id", "geometry", id_name, service_id}. Also fix the get_cql call-site comment, which still described the pre-fix REPLACE behavior. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent 3e56b62 commit afcbf62

3 files changed

Lines changed: 33 additions & 25 deletions

File tree

dataretrieval/waterdata/api.py

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3007,8 +3007,9 @@ def get_cql(
30073007

30083008
properties_list = _as_str_list(properties, "properties")
30093009

3010-
# Translate user-facing names (``daily_id``/``id``) to the wire ``id`` the
3011-
# OGC API expects, matching the typed getters.
3010+
# Drop id aliases (``daily_id``/``id``) and ``geometry`` from the wire
3011+
# ``properties`` (the feature ``id`` is always returned and renamed
3012+
# downstream), matching the typed getters.
30123013
wire_properties = _switch_properties_id(properties_list, output_id, service)
30133014

30143015
req = _construct_cql_request(

dataretrieval/waterdata/utils.py

Lines changed: 28 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -147,48 +147,54 @@ def _switch_properties_id(
147147
properties: list[str] | None, id_name: str, service: str
148148
) -> list[str]:
149149
"""
150-
Switch properties id from its package-specific identifier to the
151-
standardized "id" name that the API recognizes.
152-
153-
Replaces any service-specific id name in `properties` with "id",
154-
normalizes remaining hyphens to underscores, and drops the "geometry"
155-
and service-id entries. Returns an empty list when `properties` is empty
156-
or None.
150+
Build the wire ``properties`` list, dropping every id alias and
151+
``geometry``.
152+
153+
The feature ``id`` is always returned and is renamed to the
154+
service-specific id column (e.g. ``daily_id``) in post-processing, so
155+
it must not be requested as a property: several collections (e.g.
156+
``daily``, ``continuous``) reject ``id`` in ``properties`` with an
157+
HTTP 400. ``geometry`` is likewise excluded because it is controlled
158+
by ``skip_geometry``. Any service-specific id name (``daily_id``,
159+
``monitoring_location_id``, …) and the bare ``id`` are dropped, and
160+
remaining hyphens are normalized to underscores. Returns an empty
161+
list when `properties` is empty or None — the URL then omits the
162+
``properties`` filter and the result is shaped by :func:`_arrange_cols`.
157163
158164
Parameters
159165
----------
160166
properties : Optional[List[str]]
161167
A list containing the properties or column names to be pulled from the
162168
service, or None.
163169
id_name : str
164-
The name of the specific identifier key to look for.
170+
The service-specific id column name to drop (e.g. ``daily_id``).
165171
service : str
166172
The service name.
167173
168174
Returns
169175
-------
170176
List[str]
171-
The modified list with id names standardized to "id".
177+
The wire ``properties`` with id aliases and ``geometry`` removed
178+
and hyphens normalized.
172179
173180
Examples
174181
--------
175-
For service "monitoring-locations", it will look for
176-
"monitoring_location_id" and change
177-
it to "id".
182+
For service "daily" with ``properties=["daily_id", "value", "geometry"]``,
183+
returns ``["value"]`` — ``daily_id`` and ``geometry`` are dropped, while
184+
the ``daily_id`` column still appears in the result, renamed from the
185+
always-returned feature ``id``.
178186
"""
179187
if not properties:
180188
return []
181189
service_id = service.replace("-", "_") + "_id"
182-
last_letter = service[-1]
183-
service_id_singular = ""
184-
if last_letter == "s":
185-
service_singular = service[:-1]
186-
service_id_singular = service_singular.replace("-", "_") + "_id"
187-
# Replace id fields with "id"
188-
id_fields = [service_id, service_id_singular, id_name]
189-
properties = ["id" if p in id_fields else p.replace("-", "_") for p in properties]
190-
# Remove unwanted fields
191-
return [p for p in properties if p not in ["geometry", service_id]]
190+
# The feature ``id`` always comes back (renamed to the service id
191+
# downstream) and several collections reject it as a selectable
192+
# property; ``geometry`` is controlled by ``skip_geometry``. Drop both,
193+
# plus the service-specific id column (``id_name``) and the name derived
194+
# straight from the service (``service_id``).
195+
drop = {"id", "geometry", id_name, service_id}
196+
normalized = (p.replace("-", "_") for p in properties)
197+
return [p for p in normalized if p not in drop]
192198

193199

194200
_DATETIME_FORMATS = (

tests/waterdata_test.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -470,7 +470,8 @@ def test_get_continuous():
470470
time="2025-01-01/2025-12-31",
471471
)
472472
assert isinstance(df, DataFrame)
473-
assert "geometry" not in df.columns
473+
# ``skip_geometry`` is unset, so the server includes geometry by default.
474+
assert "geometry" in df.columns
474475
assert (
475476
df["time"].dtype.name.startswith("datetime64[")
476477
and "UTC" in df["time"].dtype.name

0 commit comments

Comments
 (0)