In order to annotate how many bytes were transferred in a given download request, the upstream log needs to emit bytes that would be parsed here.
It appears that with parse_all=True, the parser format would need to be updated to allow for the existence of the new field. Does that mean duplicating MESSAGE_V3 to a MESSAGE_V4 with the added param at the end?
|
V3_HEADER = L("download") |
|
MESSAGE_v3 = ( |
|
V3_HEADER + PIPE + REQUEST + PIPE + TLS + PIPE + PROJECT + PIPE + USER_AGENT |
|
) |
Will need to add resp.body_bytes_written to https://github.com/pypi/infra/blob/d6702898cbed406cc431a12b80270831ef292557/terraform/file-hosting/vcl/files.vcl#L589 to emit the value when this library can parse for it.
Will likely need coordination with BigQuery team to add column when the data is correctly parsed.
In order to annotate how many bytes were transferred in a given download request, the upstream log needs to emit bytes that would be parsed here.
It appears that with
parse_all=True, the parser format would need to be updated to allow for the existence of the new field. Does that mean duplicatingMESSAGE_V3to aMESSAGE_V4with the added param at the end?linehaul-cloud-function/linehaul/events/parser.py
Lines 121 to 124 in 8b8ed9d
Will need to add
resp.body_bytes_writtento https://github.com/pypi/infra/blob/d6702898cbed406cc431a12b80270831ef292557/terraform/file-hosting/vcl/files.vcl#L589 to emit the value when this library can parse for it.Will likely need coordination with BigQuery team to add column when the data is correctly parsed.