fix: pagination for null-datetime items and empty fields object

[matthewhanson]

Summary

Fixes the pagination bugs tracked for 5.0.1.

1. Null-datetime pagination (#608, #1082)

Items with datetime: null (only start_datetime/end_datetime set) broke pagination: OpenSearch sorts missing dates to the Long.MIN sentinel, which (a) loses precision when parsed in JS and (b) the lossy sort.join(',') / split(',') next token corrupted further — the follow-up page 400'd.

• next token is now base64url-encoded JSON of the sort values (round-trips numbers/nulls losslessly). buildSearchAfter still accepts the legacy comma format.
• The default sort uses missing: 0 for properties.datetime, giving missing dates a concrete, reusable search_after value instead of the sentinel.

2. fields={} strips the page-2 body

An empty fields object was serialized into the GET next link as fields={}, then re-parsed on the follow-up request as a field literally named {} — returning items with an empty _source. Empty objects/arrays are now dropped from next-link params.

#823 — already fixed

Confirmed the "next link missing when sortby field excluded" case is already resolved by the 5.0.0 sort rework (#1046); added a regression test so it stays fixed.

Tests

New tests/system/test-api-pagination.ts (null-datetime traversal + sortby-excluded). Updated two test-api-search-get assertions for the new token format. Verified: typecheck, lint, unit (69), and the pagination-affected system suite all green.

Closes #608
Closes #1082
Closes #823

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR fixes pagination edge cases in the STAC API by making next tokens lossless (base64url JSON of OpenSearch sort values), stabilizing the default datetime sort for missing datetimes, and preventing empty request params (notably fields: {}) from corrupting subsequent-page requests.

Changes:

• Encode next as base64url JSON (with legacy comma-token fallback) and adjust default sort to use missing: 0 for properties.datetime.
• Drop empty objects/arrays from pagination link params to avoid fields={} poisoning page-2 requests.
• Add/adjust system tests to cover null-datetime pagination and sortby-excluded pagination behavior.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/lib/database.ts	Updates default sort and implements base64url JSON next token parsing/encoding for search_after.
src/lib/api.ts	Filters empty values out of generated pagination link params to avoid fields={} in next links.
src/lib/types.ts	Broadens OpenSearch request typing to allow numeric/null search_after values and optional missing in sort rules.
tests/system/test-api-search-get.ts	Updates assertions to decode/verify the new base64url JSON next token format.
tests/system/test-api-pagination.ts	Adds system regressions for null-datetime pagination and sortby-excluded pagination.
CHANGELOG.md	Documents the pagination fixes.

Comments suppressed due to low confidence (1)

src/lib/api.ts:364

• For GET pagination links, fields has already been parsed into an object, and dictToURI will JSON.stringify it. On the follow-up GET request, extractFields treats fields as the GET comma syntax (not JSON), so any non-empty fields filter will be re-parsed into garbage and can strip _source on page 2. Convert the fields object back into the GET comma-separated format when building GET next links (but keep the object form for POST link bodies).

      const nextParams = pickBy(
        assign(parameters, { bbox, intersects, limit, next: lastItemSort, collections, filter }),
        isPresent
      )
      if (httpMethod === 'GET') {
        const nextQueryParameters = dictToURI(nextParams)
        link.href = `${endpoint}?${nextQueryParameters}`

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.