enhancement(transforms): Add global option to loosen ordering guarantees in stateless transforms + introduce associated metric

[ArunPiduguDD]

Summary

Introduces a new global configuration option called preserve_ordering_stateless_transforms that disables ordering guarantees in concurrently running Function Transforms. Also adds a corresponding metric estimated_concurrent_transform_scheduling_pressure to inform users when they should consider enabling this option.

Detailed Context

In Vector's existing concurrency model stateless function transforms are run concurrently (e.g. a function transform can have multiple threads working on batches of events in parallel). However, the existing implementation still guarantees event ordering (e.g. if 1000 events arrive at a transform and are processed across 10 batches/Tasks, they will still leave the transform in the same order they arrived, even if later batches complete before earlier batches).

In cases where processing latency of the events within the transform is both high & variable, then this can lead to inefficiencies - as mentioned above events that are processed in later batches can be blocked by batches/Tasks scheduled earlier (if the earlier batch is still processing when the later batch finishes)

The effect can be illustrated by this wall-time profile (measured during a benchmark test with 8 CPUs / parallel threads)

In this test the vector instance was constantly flooded with events so there are always events waiting to be processed. Multiple threads finish processing their batch, however new batches / Tasks are unable to be scheduled for these idle threads due to the fact that the head Task in the `FuturesOrdered` [queue](https://github.com/vectordotdev/vector/blob/master/src/topology/builder.rs#L1215) is still processing its batch, leading to a CPU utilization inefficiency and overall lower throughput.

When switching this to an ordered queue, the the transform is not held up by long-running tasks and the overall ingress throughput increases (graph below shows bytes / second throughput of ordered vs unordered queue - test was done using remap processor with many regex rules)

In order to determine if enabling this option is needed, this PR also adds a new metric estimated_concurrent_transform_scheduling_pressure which keeps track of how many Tasks have been completed and are blocked by the head task from being scheduled (metric is a distribution which ranges from 0-1). B/c this introduces a shared counter to the transform "hot path", ran regression benchmark tests to confirm there are no issues: https://github.com/vectordotdev/vector/actions/runs/24585515449 (note: a few tests failed to run but seems to be unrelated issues - tests also failing to run on the latest merged commit in master)

Vector configuration

Added a new regression test with the following Vector config

How did you test this PR?

Ran Vector pipelines with the preserve_ordering_stateless_transforms option set. Also ran regression tests

Change Type

• Bug fix
• New feature
• Dependencies
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

Notes

• Please read our Vector contributor resources.
• Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
• Some CI checks run only after we manually approve them.
  • We recommend adding a pre-push hook, please see this template.
  • Alternatively, we recommend running the following locally before pushing to the remote branch:
    • make fmt
    • make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
    • make test
• After a review is requested, please avoid force pushes to help us review incrementally.
  • Feel free to push as many commits as you want. They will be squashed into one before merging.
  • For example, you can run git merge origin master and git push.
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
  run make build-licenses to regenerate the license inventory and commit the changes (if any). More details here.

[pront]

Hello, thank you for this contribution! This PR can be split into two dedicated PRs, I would start with introducing estimated_concurrent_transform_scheduling_pressure first.