Source idea: HTTP API collector for Fastly Signal Science

[NeilJed]

I've been playing with the idea of using Vector to try and pull log data from Fastly Signal Science solution using their API. https://docs.fastly.com/en/ngwaf/extract-your-data

It seems that the HTTP Client source would be a good base for this but their API throws up some interesting issues to solve.

Firstly, the responses are paginated so it would need to be aware of that and look for the next page URL given in the response and automatically fetch that, eventually getting all the pages for the full data set. So I guess the the source would be making multiple HTTP requests to the API as needed.

Also the from/until timestamp - as these require UTC timestamp values they'd need to be dynamic and based on the current time that the source collection start. I was thinking about having it expressed as an negative offset from "now" in minutes. i.e from = now-10m, until = now-5m then run the source with a 5m interval. I was also thinking if now should be aligned to the nearest 5 minute to account for any delay from the desired start time and the actual time.

So yeah, I don't think this is possible out-of-the box and requires modifying the default HTTP Client source but I'd love some feed back on possible approaches or gotchas.

[jszwedko]

The sort of challenges that API has (pagination and keeping track of where to start each request) seem to fairly common across other APIs that users want to consume from Vector with. I could see enhancing the http_client source to be able to handle this. Given the scope of the change I think we'd likely want to see an RFC for it first.

As a workaround, you could write a standalone program that reads the data and passes it to Vector over one of its existing sources (like http_server).

[sonnens]

I'm dealing with this one again with yet another API so I want to revive this discussion

I don't know what the correct general way to handle pagination is, since every API is going to be different. Okta handles it with a Link header, Kandji handles it with a next link in the payload, Slack handles it with response_metadata.next_cursor in the payload

I think that maybe one solution is to add a pagination section to the http_client source and allow the user to write VRL that will generate the next URL to fetch (or null to reset to whatever the original URL is?)

For something like the Okta streaming API (just as an example. I've written the Okta source to handle this specific case) the API always returns a rel=next link, and the client is meant to pause when the payload is empty but continue processing with that next link (or, in other words, the URL to process next cycle is updated rather than reset.) I suppose something like a reset_after_pagination option or some such?

The issues here I guess is how to expose a state machine to VRL (what indicates that it should stop?) and the internal configuration (ie, the url that it should hit next time)

[sonnens]

actually I'm going to pull this is to a discussion with a better title