Currently the way of logging is showing a progress bar which is great if you have it running it from CLI but I have it as a job running on K8S.
Are you open for getting a PR to add an additional method of logging next to the tqdm, The default will be the same and only if supplied --log-format json it would use the alternate logging parser.
I would be adding a json logger which would log the following
For start event
- type (starting, processing, completed)
- entity (artist, release ect)
- total
- timestamp
- message
For update event
- type (starting, processing, completed)
- entity (artist, release ect)
- processed
- total
- percentage
- rate
- elapsed_seconds
- timestamp
- message
For completed event
- type (starting, processing, completed)
- entity (artist, release ect)
- processed
- total
- elapsed_seconds
- timestamp
- message
Logs would look like the following (I dropped most of the logs obviously from this snippet)
{"type":"starting","entity":"artist","total":9772142,"timestamp":1763464789.3201, "message":"Started processing entity artist"}
{"type":"progressing","entity":"artist","processed":187866,"total":9772142,"percentage":1.92,"rate":37563.22,"elapsed_seconds":5.0,"timestamp":1763464794.321428, "message":"Processing entity artist"}
{"type":"progressing","entity":"artist","processed":408185,"total":9772142,"percentage":4.18,"rate":40813.07,"elapsed_seconds":10.0,"timestamp":1763464799.321429, "message":"Processing entity artist"}
{"type":"progressing","entity":"artist","processed":9792439,"total":9772142,"percentage":100.21,"rate":75227.58,"elapsed_seconds":130.17,"timestamp":1763464919.4909692, "message":"Processing entity artist"}
{"type":"completed","entity":"artist","processed":9798352,"total":9772142,"elapsed_seconds":130.21,"timestamp":1763464919.5340512,"message":"Completed processing entity artist"}
Currently the way of logging is showing a progress bar which is great if you have it running it from CLI but I have it as a job running on K8S.
Are you open for getting a PR to add an additional method of logging next to the tqdm, The default will be the same and only if supplied
--log-format jsonit would use the alternate logging parser.I would be adding a json logger which would log the following
For start event
For update event
For completed event
Logs would look like the following (I dropped most of the logs obviously from this snippet)
{"type":"starting","entity":"artist","total":9772142,"timestamp":1763464789.3201, "message":"Started processing entity artist"} {"type":"progressing","entity":"artist","processed":187866,"total":9772142,"percentage":1.92,"rate":37563.22,"elapsed_seconds":5.0,"timestamp":1763464794.321428, "message":"Processing entity artist"} {"type":"progressing","entity":"artist","processed":408185,"total":9772142,"percentage":4.18,"rate":40813.07,"elapsed_seconds":10.0,"timestamp":1763464799.321429, "message":"Processing entity artist"} {"type":"progressing","entity":"artist","processed":9792439,"total":9772142,"percentage":100.21,"rate":75227.58,"elapsed_seconds":130.17,"timestamp":1763464919.4909692, "message":"Processing entity artist"} {"type":"completed","entity":"artist","processed":9798352,"total":9772142,"elapsed_seconds":130.21,"timestamp":1763464919.5340512,"message":"Completed processing entity artist"}