Skip to content

[logarchive] completely redesign the logarchive parser for performance #237

Description

@cvandeplas
  • rewrite the parser completely (from scratch) , remove many layers and functions that make code flow difficult to read

  • unifiedlogs_iterator stdout reading kill everything, CPU is only 25%.

  • Let unifiedlogs save to a temp file, read the temp file while the process is still running and throw these lines into batches of lines to process

  • write batches of records to the output file

  • use orjson to improve performance loading and generating/saving json

  • do some more tuning here and there

  • work on unifiedlogs_iterator multithreaded, and initiate a PR to that project.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions