Pagination links not included if sortby field is excluded from search results

[JackDunnNZ]

When searching, if the fields parameter causes a field used in the sortby parameter to be excluded from the results, it seems that the server does not include the next link for pagination.

As an example using earth search, here is a request that excludes the field used for sorting:

curl 'https://earth-search.aws.element84.com/v1/search' \
  -X POST \
  -H 'Content-Type: application/json' \
  --data-raw '{
    "limit": 1,
    "collections": ["sentinel-2-c1-l2a"],
    "sortby": [{"field": "properties.updated", "direction": "desc"}],
    "fields": {"exclude": ["links", "assets"], "include": []}
  }'

which gives

{"type":"FeatureCollection","stac_version":"1.0.0","stac_extensions":[],"context":{"limit":1,"matched":19905986,"returned":1},"numberMatched":19905986,"numberReturned":1,"features":[{"stac_version":"1.0.0","bbox":[-43.437734,78.265531,-42.757435,78.348647],"geometry":{"coordinates":[[[-43.437733510087924,78.33900109809352],[-43.410287878150164,78.26553138490355],[-42.75743544280556,78.34864725204903],[-43.437733510087924,78.33900109809352]]],"type":"Polygon"},"id":"S2A_T24XVM_20190430T170855_L2A","collection":"sentinel-2-c1-l2a","type":"Feature","properties":{"datetime":"2019-04-30T17:10:39.959000Z"}}],"links":[{"rel":"root","type":"application/json","href":"https://earth-search.aws.element84.com/v1"}]}

If the properties.updated field is specifically included, then the next link is included in the response.

curl 'https://earth-search.aws.element84.com/v1/search' \
  -X POST \
  -H 'Content-Type: application/json' \
  --data-raw '{
    "limit": 1,
    "collections": ["sentinel-2-c1-l2a"],
    "sortby": [{"field": "properties.updated", "direction": "desc"}],
    "fields": {"exclude": ["links", "assets"], "include": ["properties.updated"]}
  }'

giving

{"type":"FeatureCollection","stac_version":"1.0.0","stac_extensions":[],"context":{"limit":1,"matched":19906000,"returned":1},"numberMatched":19906000,"numberReturned":1,"features":[{"stac_version":"1.0.0","bbox":[-100.889805,17.211145,-100.481581,18.083373],"geometry":{"coordinates":[[[-100.88980499375167,18.079807152724634],[-100.88127718540649,17.261739081673692],[-100.69294676442333,17.211145110345363],[-100.48158122778172,18.083373358751057],[-100.88980499375167,18.079807152724634]]],"type":"Polygon"},"id":"S2A_T14QLE_20190430T171850_L2A","collection":"sentinel-2-c1-l2a","type":"Feature","properties":{"datetime":"2019-04-30T17:28:58.911000Z","updated":"2024-10-02T18:34:48.587Z"}}],"links":[{"rel":"next","title":"Next page of Items","method":"POST","type":"application/geo+json","href":"https://earth-search.aws.element84.com/v1/search","merge":false,"body":{"sortby":[{"field":"properties.updated","direction":"desc"}],"fields":{"exclude":["links","assets"],"include":["properties.updated"]},"collections":["sentinel-2-c1-l2a"],"limit":1,"next":"2024-10-02T18:34:48.587Z"}},{"rel":"root","type":"application/json","href":"https://earth-search.aws.element84.com/v1"}]}

It is easy to workaround by making sure the sorting fields are included, but it seems like a hidden pitfall since a client may simply stop iterating with no indication that the search results are incomplete.