Currently, the request to proofpoint is returned as a blob and then the next stage parses over the entire parsed json response structure streaming out reputations as they are detected.
This has a level (albeit small) of latency (while the response is streamed to the response buffer - plus, depending on how request works, may also then parse the response buffer) and a memory overhead as the entire parsed json structure resides in memory to be passed from the request response handler to the reputation searching component.
An alternate approach is to stream the response from the request directly to a json lexical streaming parser, and based on the known structure of the proofpoint response, start processing (and discarding) each file reputation as it is hydrated from the stream.
This removes both the memory and latency overheads at the client end (though doesn't overcome the server side latency, it's the best we can do).
Currently, the request to proofpoint is returned as a blob and then the next stage parses over the entire parsed json response structure streaming out reputations as they are detected.
This has a level (albeit small) of latency (while the response is streamed to the response buffer - plus, depending on how request works, may also then parse the response buffer) and a memory overhead as the entire parsed json structure resides in memory to be passed from the request response handler to the reputation searching component.
An alternate approach is to stream the response from the request directly to a json lexical streaming parser, and based on the known structure of the proofpoint response, start processing (and discarding) each file reputation as it is hydrated from the stream.
This removes both the memory and latency overheads at the client end (though doesn't overcome the server side latency, it's the best we can do).