[Detail Bug] List API: pagination validation rejects start_after values before prefix

[detail-app]

Detail Bug Report

https://app.detail.dev/org_89d327b3-b883-4365-b6a3-46b6701342a9/bugs/bug_5977fc32-bd19-4d12-a07b-da25a6780061

Introduced in 8a33e9b by @shikhar on Nov 7, 2025

Summary

• Context: The ListItemsRequest validation in common/src/types/resources.rs validates pagination parameters for listing operations (basins, streams, access tokens).
• Bug: The validation rejects queries where start_after < prefix lexicographically, violating industry-standard API semantics established by AWS S3, Google Cloud Storage, and Azure Blob Storage.
• Actual vs. expected: The validation rejects valid queries that all major cloud storage APIs accept.
• Impact: Users cannot implement common pagination patterns involving prefix filters, breaking compatibility with established API conventions.

Code with Bug

impl<P, S> TryFrom<ListItemsRequestParts<P, S>> for ListItemsRequest<P, S>
where
    P: Deref<Target = str> + Default,
    S: Deref<Target = str> + Default,
{
    type Error = StartAfterLessThanPrefixError;

    fn try_from(parts: ListItemsRequestParts<P, S>) -> Result<Self, Self::Error> {
        let start_after: &str = &parts.start_after;
        let prefix: &str = &parts.prefix;

        if !start_after.is_empty() && !prefix.is_empty() && start_after < prefix {
            return Err(StartAfterLessThanPrefixError);  // <-- BUG 🔴 rejects valid (start_after, prefix) combinations
        }

        Ok(Self(parts))
    }
}

Explanation

Cloud storage list APIs commonly apply prefix and start_after/startOffset independently; there is no requirement that start_after be lexicographically >= prefix. AWS S3 explicitly states “StartAfter can be any key in the bucket”.

The repo’s backend already implements the expected semantics by taking max(prefix_start, start_after_key) when building the range, meaning start_after < prefix is handled safely (it effectively behaves like just prefix for that case), while still allowing start_after to affect results when it falls within the prefix range. The API validation layer therefore rejects inputs the backend can correctly process, creating an API/backend inconsistency and blocking a common pagination workflow (reusing a prior page token/marker while adding a prefix filter).

Codebase Inconsistency

Backend range construction accepts start_after < prefix:

pub fn ser_key_range(prefix: &BasinNamePrefix, start_after: &BasinNameStartAfter) -> Range<Bytes> {
    let prefix_start = ser_key_prefix(prefix);
    let start = if !start_after.is_empty() {
        let start_after_key = ser_key_start_after(start_after);
        std::cmp::max(prefix_start, start_after_key)  // <-- accepts start_after before prefix
    } else {
        prefix_start
    };
    let end = ser_key_prefix_end(prefix);
    start..end
}

Recommended Fix

Remove the start_after < prefix validation by changing ListItemsRequest construction to infallible (e.g., TryFrom -> From), and update API docs/OpenAPI + SDK tests to reflect that prefix and start_after are applied independently.

History

This bug was introduced in commit 8a33e9b. The initial commit of the project included the StartAfterLessThanPrefixError validation which rejects queries where start_after < prefix, diverging from AWS S3's established API semantics. The validation was likely based on an assumption that such queries are invalid, but AWS S3 explicitly documents that "StartAfter can be any key in the bucket" with no restriction relative to prefix.

[shikhar]

Not really a bug, but I can see how we may as well just allow it.