- rewrite the parser completely (from scratch) , remove many layers and functions that make code flow difficult to read - unifiedlogs_iterator stdout reading kill everything, CPU is only 25%. - Let unifiedlogs save to a temp file, read the temp file while the process is still running and throw these lines into batches of lines to process - write batches of records to the output file - use orjson to improve performance loading and generating/saving json - do some more tuning here and there - work on unifiedlogs_iterator multithreaded, and initiate a PR to that project.
rewrite the parser completely (from scratch) , remove many layers and functions that make code flow difficult to read
unifiedlogs_iterator stdout reading kill everything, CPU is only 25%.
Let unifiedlogs save to a temp file, read the temp file while the process is still running and throw these lines into batches of lines to process
write batches of records to the output file
use orjson to improve performance loading and generating/saving json
do some more tuning here and there
work on unifiedlogs_iterator multithreaded, and initiate a PR to that project.