Skip to content

Adapt hybrid data pipeline #72

Description

@lherman-cs

Currently, the pipeline is driven by async events. This is great for latency and CPU usage in idle/low workload cases. But when the pipeline is handling high throughput, the event synchronization overhead becomes significant, taking 10-15% of the CPU.

Luckily, Linux 6.1+ has done something similar in NAPI: Kernel Polling with Interrupt Suppression in Linux 6.1.

More details: https://www.linkedin.com/pulse/napi-new-api-linux-kernel-networking-david-zhu-wcz6c/

Image

Instead of being pure event-driven, the loop can poll until the budget is hit.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Status
    Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions