fix: bump multistore to 0.6.4 (multipart SignatureDoesNotMatch on keys with =) (#181)

Fixes #180.

Multipart uploads to keys containing `=` (Hive-style partition paths
like `by_country/country_iso=ETH/ETH.pmtiles`) fail with
`SignatureDoesNotMatch`: multistore 0.6.3's `build_backend_url` splices
the raw key into the backend URL for raw-signed multipart operations,
while AWS/MinIO reconstruct the canonical URI by strict-encoding the
decoded path — so the signatures diverge. Fixed upstream in
developmentseed/multistore#105, released as 0.6.4.

## History

1. **Commit 1 (test only)** — pinned the expected SigV4 strict encoding
locally; CI failed on it against 0.6.3, demonstrating the regression
(red).
2. **Commit 2 (dependency bump)** — bumps all multistore crates to
0.6.4; CI green.
3. **Deconflicted with main after #183**, and the local pins were
**dropped in favor of the upstream tests**: the 0.6.4 release itself
ships seven equivalent-or-stronger tests in the multipart module
(including the exact `country_iso=ETH` case, upload-id encoding, and
literal-`%3D` double-encoding). Keeping a local copy would test the
dependency, not this repo.
4. **Worker-level multipart coverage added** (tests/test_writes.py),
layered by what each environment can execute:
- anonymous `CreateMultipartUpload` on a hive key → 403 at the gate —
hermetic, every run (verified against a local 0.6.4 worker);
- authenticated create on the probe's unassumable role → bounded,
parseable fail-closed error — runs on same-repo CI (exercises multipart
routing through the write-authz + federation seam);
- **full round-trip** (create → 5 MiB + 1 KiB parts → complete →
read-back → delete) on `by_country/country_iso=ETH/…` — the proxy-level
#180 guarantee. Skips until the write-probe infra exists; runs against
any provisioned worker via env vars.

## Activating the round-trip (follow-up, tracked with #183's write-probe
work)

Probe bucket + IAM role, a dedicated CI signing key whose public JWKS is
served at an issuer URL the role's OIDC provider trusts (not staging's
key), the stub serving the real bucket/role instead of placeholders, and
ci.yml exporting `CI_WRITE_PROBE_ROLE_ARN`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

## Behavior changes shipped by the bump (review-verified against the
vendored sources)

- **Special-character key addressing changes for the presigned CRUD
path.** 0.6.3's `Path::from` percent-encoded `* % ~ #`-class characters
into the *stored* object name; 0.6.4's `Path::parse` is byte-faithful.
Any object PUT through the proxy since writes shipped (2026-06-23) with
such a key was stored under a mangled name and becomes unreachable by
its original key after this deploys. Exposure is a ~3-week window of
simple presigned PUTs; worth a quick bucket scan for `%`-containing
names before/after deploy.
- **Degenerate keys now 400 instead of being silently normalized.**
0.6.4 adds `validate_key` on every keyed operation (reads included):
leading/trailing slashes (folder-marker probes), empty `//` segments,
`.`/`..` segments, and control bytes return 400 `InvalidRequest` where
0.6.3 normalized them (`GET a//b` served `a/b`; `PUT dir/` wrote `dir`).
Upstream frames this as deliberate strictness; noting it here since it's
invisible in this diff.

(A claimed quick-xml 0.37→0.41 serialization drift was refuted by
byte-level A/B comparison — output is identical; backend XML parsing
goes through object_store's separately-pinned quick-xml, unchanged by
this PR.)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Author: alukach

Cargo.lock

@@ -36,10 +36,10 @@ path = "tests/fixtures.rs"
 
 [dependencies]
 # Multistore
-multistore = { version = "0.6.3", features = ["azure"] }
-multistore-oidc-provider = "0.6.3"
-multistore-path-mapping = "0.6.3"
-multistore-sts = "0.6.3"
+multistore = { version = "0.6.4", features = ["azure"] }
+multistore-oidc-provider = "0.6.4"
+multistore-path-mapping = "0.6.4"
+multistore-sts = "0.6.4"
 
 # Serialization
 serde = { version = "1", features = ["derive"] }
@@ -61,7 +61,7 @@ tracing = "0.1"
 # Pulled in transitively via object_store -> rand. getrandom doesn't support
 # wasm32-unknown-unknown unless the `wasm_js` backend is enabled, so opt in here.
 getrandom = { version = "0.4", features = ["wasm_js"] }
-multistore-cf-workers = { version = "0.6.3", features = ["azure"] }
+multistore-cf-workers = { version = "0.6.4", features = ["azure"] }
 # On wasm32-unknown-unknown reqwest selects its browser `fetch` backend by
 # target arch (not a cargo feature), so the default TLS/backend features are
 # unnecessary here. We only use `Client::new()`, `.send()` and `.text()`, none

Cargo.toml

@@ -22,6 +22,7 @@
 """
 
 import json
+import os
 from http.server import BaseHTTPRequestHandler, HTTPServer
 from pathlib import Path
 
@@ -57,6 +58,17 @@ def _fixture(name):
 
 WRITE_PRODUCT_JSON = _fixture("product_write_probe")
 
+WRITE_CONNECTION_JSON = _fixture("data_connection_write_probe")
+# Activation plumbing: the same CI_WRITE_PROBE_* variables that un-skip
+# test_multipart_roundtrip_hive_partition replace the fixture placeholders
+# here, so the test gate and the served connection can't drift apart.
+_details = WRITE_CONNECTION_JSON["details"]
+_details["bucket"] = os.environ.get("CI_WRITE_PROBE_BUCKET", _details["bucket"])
+_details["region"] = os.environ.get("CI_WRITE_PROBE_REGION", _details["region"])
+WRITE_CONNECTION_JSON["authentication"]["role_arn"] = os.environ.get(
+    "CI_WRITE_PROBE_ROLE_ARN", WRITE_CONNECTION_JSON["authentication"]["role_arn"]
+)
+
 # ── Failure probes ─────────────────────────────────────────────────
 # Synthetic products that make the control plane misbehave on purpose, so the
 # proxy's fail-closed error mapping is exercised in CI (the original incident
@@ -81,9 +93,7 @@ def _fixture(name):
     f"/api/v1/products/{WRITE_ACCOUNT}": {"products": [WRITE_PRODUCT_JSON]},
     f"/api/v1/products/{WRITE_ACCOUNT}/{WRITE_PRODUCT}": WRITE_PRODUCT_JSON,
     f"/api/v1/products/{WRITE_ACCOUNT}/{WRITE_PRODUCT}/permissions": ["read", "write"],
-    f"/api/v1/data-connections/{WRITE_CONNECTION}": _fixture(
-        "data_connection_write_probe"
-    ),
+    f"/api/v1/data-connections/{WRITE_CONNECTION}": WRITE_CONNECTION_JSON,
 }
 
 # Import-time drift guards: routes are keyed on the constants above, bodies

tests/stub_api.py

@@ -25,7 +25,9 @@
 smoke tests (tests/test_federation.py, wired into staging.yml).
 """
 
+import functools
 import os
+import uuid
 import xml.etree.ElementTree as ET
 
 import pytest
@@ -63,8 +65,13 @@ def sts_exchange(token):
     )
 
 
+@functools.lru_cache(maxsize=1)
 def exchange_token():
-    """POST /.sts and return (access_key_id, secret_access_key, session_token)."""
+    """POST /.sts and return (access_key_id, secret_access_key, session_token).
+
+    Cached: one real exchange serves every positive-path test instead of ~8
+    identical OIDC verifications per run (the negatives call sts_exchange
+    directly and stay uncached)."""
     resp = sts_exchange(ID_TOKEN)
     assert resp.status_code == 200, f"/.sts exchange failed ({resp.status_code}): {resp.text[:300]}"
     # Match by local name to stay independent of the response's xmlns.
@@ -75,9 +82,10 @@ def exchange_token():
     return fields["AccessKeyId"], fields["SecretAccessKey"], fields["SessionToken"]
 
 
-def s3_client(session_token=None):
+def s3_client(session_token=None, **config_kwargs):
     """Credentialed S3 client via /.sts; `session_token` overrides the sealed
-    token so tests can present corrupted ones."""
+    token so tests can present corrupted ones. `config_kwargs` merge into the
+    botocore Config (e.g. retries)."""
     import boto3  # deferred: only the credentialed tests need it
     from botocore.config import Config
 
@@ -89,16 +97,102 @@ def s3_client(session_token=None):
         aws_secret_access_key=secret_key,
         aws_session_token=session_token or real_token,
         region_name="us-east-1",
-        config=Config(s3={"addressing_style": "path"}),
+        config=Config(s3={"addressing_style": "path"}, **config_kwargs),
     )
 
 
-def test_anonymous_write_to_federated_product_denied():
-    """No credentials -> denied at the gate, before any backend/STS call."""
-    resp = requests.put(f"{PROXY_URL}/{WRITE_ACCOUNT}/{WRITE_PRODUCT}/denied.txt")
+# Hive-style partition key (`country_iso=ETH`): the raw-signed multipart path
+# where the 0.6.3 encoding bug (#180) lived. Only the round-trip test below
+# reaches that encoding seam — the denial and fail-closed layers die at the
+# authz gate / federation first; their hive keys just keep the paths uniform.
+HIVE_KEY_DIR = "by_country/country_iso=ETH"
+
+
+@pytest.mark.parametrize(
+    "method,path",
+    [
+        ("put", "denied.txt"),
+        ("post", f"{HIVE_KEY_DIR}/denied.pmtiles?uploads"),
+    ],
+    ids=["put", "multipart-create"],
+)
+def test_anonymous_write_to_federated_product_denied(method, path):
+    """No credentials -> denied at the gate, before any backend/STS call.
+    Every write op shares the gate; multipart create is one more of them."""
+    resp = getattr(requests, method)(
+        f"{PROXY_URL}/{WRITE_ACCOUNT}/{WRITE_PRODUCT}/{path}"
+    )
     assert resp.status_code == 403
 
 
+needs_probe_infra = pytest.mark.skipif(
+    not os.environ.get("CI_WRITE_PROBE_ROLE_ARN"),
+    reason="write-probe AWS infra not configured (set CI_WRITE_PROBE_ROLE_ARN)",
+)
+
+
+@needs_token
+@needs_probe_infra
+def test_multipart_roundtrip_hive_partition():
+    """The proxy-level #180 guarantee: multipart create/upload/complete on a
+    hive-partitioned key, read back and verified — the only test that reaches
+    the raw-signed backend-URL encoding seam end-to-end.
+
+    Activation (see #183's follow-ups): the probe bucket + IAM role, a worker
+    signing key whose issuer the role's OIDC provider trusts (a dedicated CI
+    key — not staging's), and ci.yml exporting the CI_WRITE_PROBE_* vars to
+    both the stub and pytest. The stub derives its served bucket/role from
+    the same variables (see stub_api.py), so this gate and the served config
+    can't drift apart.
+    """
+    client = s3_client()
+    key = (
+        f"{WRITE_PRODUCT}/{HIVE_KEY_DIR}/"
+        f"gh-{os.environ.get('GITHUB_RUN_ID', 'local')}-{uuid.uuid4().hex}.pmtiles"
+    )
+    bodies = (b"a" * (5 * 1024 * 1024), b"b" * 1024)  # 5 MiB min non-final part
+
+    upload_id = client.create_multipart_upload(Bucket=WRITE_ACCOUNT, Key=key)[
+        "UploadId"
+    ]
+    try:
+        parts = []
+        for n, body in enumerate(bodies, 1):
+            resp = client.upload_part(
+                Bucket=WRITE_ACCOUNT, Key=key, UploadId=upload_id,
+                PartNumber=n, Body=body,
+            )
+            # Carry the Checksum* members through. Not required today (no
+            # ChecksumAlgorithm was declared at create, so S3 validates only
+            # PartNumber+ETag), but the moment one is declared — e.g. by
+            # switching to boto3's transfer manager — S3 rejects a Complete
+            # that omits per-part checksums. Harmless now, load-bearing then.
+            parts.append(
+                {"PartNumber": n, "ETag": resp["ETag"]}
+                | {k: v for k, v in resp.items() if k.startswith("Checksum")}
+            )
+        client.complete_multipart_upload(
+            Bucket=WRITE_ACCOUNT, Key=key, UploadId=upload_id,
+            MultipartUpload={"Parts": parts},
+        )
+        got = client.get_object(Bucket=WRITE_ACCOUNT, Key=key)["Body"].read()
+        assert got == b"".join(bodies)
+    finally:
+        # Best effort, order-independent: delete covers a completed object
+        # (even one completed server-side after a client-side error), abort
+        # covers an unfinished upload; the bucket lifecycle rule backstops.
+        for cleanup in (
+            lambda: client.delete_object(Bucket=WRITE_ACCOUNT, Key=key),
+            lambda: client.abort_multipart_upload(
+                Bucket=WRITE_ACCOUNT, Key=key, UploadId=upload_id
+            ),
+        ):
+            try:
+                cleanup()
+            except Exception:
+                pass
+
+
 @needs_token
 def test_sts_exchange_issues_credentials():
     """The proxy verifies the GitHub OIDC token (issuer discovery, aud check)
@@ -186,21 +280,36 @@ def test_signed_request_with_special_char_key_verifies():
 
 
 @needs_token
-def test_federated_write_fails_closed():
+@pytest.mark.parametrize("op", ["put_object", "create_multipart_upload"])
+def test_federated_write_fails_closed(op):
     """The write probe's role is unassumable by design (placeholder ARN,
     throwaway signing key): backend STS federation must fail with a bounded,
     parseable S3 error — never a hang, never a silent success. This is the
-    incident class that motivated the stub, kept permanently exercised."""
+    incident class that motivated the stub, kept permanently exercised.
+    Federation errors before any backend URL is built, so this pins the
+    fail-closed seam for every write op; key encoding is the round-trip
+    test's job."""
     import botocore.exceptions
 
-    client = s3_client()
+    # max_attempts=1: botocore's default mode retries the 502/503 this
+    # produces, multiplying live AWS STS round-trips for no extra signal.
+    client = s3_client(retries={"max_attempts": 1})
+    kwargs = {
+        "Bucket": WRITE_ACCOUNT,
+        "Key": f"{WRITE_PRODUCT}/{HIVE_KEY_DIR}/fail-closed.pmtiles",
+    }
+    if op == "put_object":
+        kwargs["Body"] = b"x"
     with pytest.raises(botocore.exceptions.ClientError) as exc:
-        client.put_object(
-            Bucket=WRITE_ACCOUNT, Key=f"{WRITE_PRODUCT}/fail-closed.txt", Body=b"x"
-        )
-    # The exact code depends on how AWS STS rejects the assertion; what
-    # matters is that boto3 could parse an S3-shaped error at all.
-    assert exc.value.response["Error"].get("Code"), "unparseable error response"
+        getattr(client, op)(**kwargs)
+    code = exc.value.response["Error"].get("Code")
+    assert code, "unparseable error response"
+    # Federation failure maps only to AccessDenied / BackendAuthenticationFailed
+    # / ServiceUnavailable / InternalError; the inbound-auth codes would mean
+    # verification broke before federation was even attempted.
+    assert code not in {"SignatureDoesNotMatch", "InvalidRequest"}, (
+        f"{code}: rejected before the federation seam this test pins"
+    )
 
 
 @needs_token