Redirected-to URLs cannot be queried in page list

[SuaYoo]

Issue

When a seed redirects, the redirect target URL cannot be searched for or queried in the page list. This leads to UX issues downstream in tools in ReplayWeb.Page.

For example, given the seed URL https://iana.org/domains/example, the page list will return a url of https://iana.org/domains/example. During replay, the actual URL displayed for the archived page will be https://www.iana.org/help/example-domains. This would indicate that the URL exists in the page list; however, searching through the page list for https://www.iana.org/help/example-domains does not return any results.

Requirements

The page list should include the final URL that returns a 200 for redirects. This URL should be searchable/able to be queried from the page list endpoint.

Ideas

Some suggestions for ways to approach the issue:

• Include the redirected-to URL as a separate non-seed URL entry in the pages list.
• Add a new field to the page entry that keeps track of URLs in the redirect chain, and allow querying by them.

[tw4l]

We should probably go with the second option and add a new field that can keep track of the redirect chain URLs, since adding these as separate entries in the page list would affect the page count in Browsertrix and ReplayWeb.page, which might be confusing for users.