Skip to content

[Core] Avoid O(n^2) header dedup when cloning EF_UNIQ_HEADERS events#3061

Open
celliso1 wants to merge 5 commits into
signalwire:masterfrom
celliso1:events-avoid-on2-header-dedup-on-clone
Open

[Core] Avoid O(n^2) header dedup when cloning EF_UNIQ_HEADERS events#3061
celliso1 wants to merge 5 commits into
signalwire:masterfrom
celliso1:events-avoid-on2-header-dedup-on-clone

Conversation

@celliso1

@celliso1 celliso1 commented Jun 28, 2026

Copy link
Copy Markdown

Description

Problem

switch_event_dup() clones a source event by re-adding every header with
switch_event_add_header_string(). When the source has EF_UNIQ_HEADERS
set — which is the case for SWITCH_EVENT_CHANNEL_DATA,
SWITCH_EVENT_REQUEST_PARAMS and SWITCH_EVENT_MESSAGE, i.e. the events
used to hold a channel's variable list — the destination inherits the
flag, so switch_event_base_add_header() calls switch_event_del_header()
(a full linear scan of the list built so far) on every add to enforce
uniqueness. Duplicating an N-header event is therefore O(n²).

The scan is wasted work: the source list is already unique (uniqueness was
enforced when each header was first inserted), so copying it can never
create a duplicate, with or without the per-add scan.

Why it matters

switch_event_dup() backs switch_core_get_variables() and
switch_channel_get_variables(), which clone an EF_UNIQ_HEADERS
variable event. Any workload that duplicates large EF_UNIQ_HEADERS
events — channel-variable snapshots, or ESL consumers cloning big
events — pays this O(n²) on every dup.

On a production voicemail box carrying ~191 channel variables per call,
switch_event_del_header_val() (the per-add uniqueness scan) is 47–69%
of FS CPU in perf, and call-graph profiles show a substantial share
reached through execute_on -> *_get_variables -> switch_event_dup — the
path this patch fixes.

Scope: this patch fixes the dup path only. del_header_val is also
reached via switch_event_merge() (also in execute_on) and per-variable
switch_channel_set_variable(); those are separate O(n²) paths this
change does not address (future work: avoid the dup+merge in execute_on,
or hash-index event headers). The exact prod split between these callers
is not yet quantified.

Fix

Clear EF_UNIQ_HEADERS on the destination for the duration of the copy
loop in switch_event_dup(), then restore it. ~10 lines, no API change.

Microbench

tests/unit/switch_event.c (dup_uniq_bench), dup of an N-header
CHANNEL_DATA event, 4000 iterations, time per dup:

N before after speedup
50 4.80 µs 3.19 µs 1.5×
100 11.34 µs 5.88 µs 1.9×
191 28.29 µs 10.99 µs 2.6×
400 107.05 µs 23.27 µs 4.6×

Real-call validation

Validated on production voicemail hosts under real deposit load — across
CPU, hypervisor, and FreeSWITCH version.
Same-node, before vs after
swapping in the patched lib (40 s perf windows, self-time):

environment del_header_val stock patched reduction
VMware / Intel / 1.10.12 56.7% of FS CPU 8.5% −85%
VMware / AMD / 1.10.12 51.4% 7.9% −85%
Proxmox / EPYC / 1.11.1 ~47% 6.4% −86%

A consistent ~85% reduction in the per-add uniqueness scan across all
three environments — the cache-bound Intel box (highest stock cost) drops
the most in absolute terms, −48 points of FS CPU. The call-graph
confirms the mechanism: with the patch the
switch_channel_execute_on → *_get_variables → switch_event_dup chains no
longer reach del_header_val; the ~6–8% residual is a separate path
(switch_event_merge, also reached from execute_on) that this change
deliberately does not touch.

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update
  • Code cleanup / refactor

Testing

Profiled on production voicemail deposit service.

  • Added/updated unit tests
  • Tested manually
  • Tested with live SignalWire credentials (if applicable)

Checklist

  • I have read the CONTRIBUTING guidelines
  • My code follows the project's style guidelines
  • I have added tests for my changes (if applicable)
  • I have updated documentation (if applicable)
  • All existing tests pass

@celliso1

Copy link
Copy Markdown
Author

@andywolk — would you have a moment to look at this? It's a self-contained ~10-line hot-path fix (O(n^2) -> O(n) in switch_event_dup() for EF_UNIQ_HEADERS events) with unit + regression tests.

Since opening it I validated the change on a production voicemail fleet — same-node before/after under real deposit load — and the win holds across CPU, hypervisor, and FreeSWITCH version:

environment del_header_val stock patched
VMware / Intel / 1.10.12 56.7% 8.5%
VMware / AMD / 1.10.12 51.4% 7.9%
Proxmox / EPYC / 1.11.1 ~47% 6.4%

A consistent ~85% reduction in the per-add uniqueness scan (also summarized in the PR body). Happy to adjust anything — thanks!

@celliso1
celliso1 force-pushed the events-avoid-on2-header-dedup-on-clone branch from d2e9057 to 5a8505d Compare June 28, 2026 16:52
Comment thread tests/unit/switch_event.c
@celliso1
celliso1 requested a review from morbit85 June 29, 2026 15:21
Comment thread src/switch_event.c Outdated
Comment thread tests/unit/switch_event.c Outdated
@celliso1
celliso1 requested a review from andywolk June 29, 2026 15:49
@andywolk andywolk changed the title events: avoid O(n^2) header dedup when cloning EF_UNIQ_HEADERS events [Core] Events: avoid O(n^2) header dedup when cloning EF_UNIQ_HEADERS events Jun 29, 2026
@andywolk andywolk changed the title [Core] Events: avoid O(n^2) header dedup when cloning EF_UNIQ_HEADERS events [Core] Avoid O(n^2) header dedup when cloning EF_UNIQ_HEADERS events Jun 29, 2026
@andywolk andywolk added the bug Something isn't working label Jun 30, 2026
@andywolk
andywolk force-pushed the events-avoid-on2-header-dedup-on-clone branch from 7cac871 to 1002510 Compare July 1, 2026 13:31
@andywolk
andywolk force-pushed the events-avoid-on2-header-dedup-on-clone branch from 1002510 to 0f04cf5 Compare July 4, 2026 13:37
switch_event_dup() copies a source event header by header via
switch_event_add_header_string(). When the source carries
EF_UNIQ_HEADERS (SWITCH_EVENT_CHANNEL_DATA / REQUEST_PARAMS / MESSAGE --
e.g. a channel's variable list), the destination inherits the flag, so
switch_event_base_add_header() runs a full switch_event_del_header()
linear scan of the partially-built list on every add to enforce
uniqueness. Cloning an N-header event is therefore O(n^2).

That scan is redundant while cloning: the source already enforced
uniqueness on insert, so copying it cannot introduce duplicates
regardless of the per-add scan. Clear EF_UNIQ_HEADERS on the
destination for the duration of the copy loop, then restore it.

This is a hot path. switch_channel_execute_on() -- invoked on ring,
answer, media, transfer, park, record and playback -- calls
switch_core_get_variables() and switch_channel_get_variables(), each of
which dup()s an EF_UNIQ_HEADERS event. On a production voicemail server
carrying ~191 channel variables per call, switch_event_del_header_val()
accounted for 47-69% of all FreeSWITCH CPU time in on-box perf profiles.

Microbenchmark (tests/unit/switch_event.c, switch_event_dup of an
N-header CHANNEL_DATA event, 4000 iterations, time per dup):

    N      before     after
    50     4.80 us    3.19 us
    100   11.34 us    5.88 us
    191   28.29 us   10.99 us   (2.6x)
    400  107.05 us   23.27 us   (4.6x)

"before" scales ~O(n^2); "after" is linear. Behaviour is unchanged: the
clone has identical headers, values and order, keeps EF_UNIQ_HEADERS,
and still enforces uniqueness on subsequent adds (covered by the test).

Signed-off-by: Calvin Ellison <cellison@youmail.com>
Adds a dup_uniq_bench case to the switch_event unit test. For an
EF_UNIQ_HEADERS (CHANNEL_DATA) source event it verifies the clone
preserves header count, values and the EF_UNIQ_HEADERS flag, and that
re-setting an existing key does not produce a duplicate in the clone.
It also prints a switch_event_dup timing sweep across header counts,
used to validate the O(n^2) -> O(n) change in switch_event_dup().

Signed-off-by: Calvin Ellison <cellison@youmail.com>
Covers the EF_UNIQ_HEADERS edge case for the O(n^2)->O(n) dup change: a well-formed EF_UNIQ source stays unique through dup, and a malformed source already holding duplicate names (only possible if headers were added before the flag was set) is copied faithfully rather than silently collapsed to the last value.

Signed-off-by: Calvin Ellison <cellison@youmail.com>
The dup_uniq_bench timing sweep is a benchmark rather than a pass/fail correctness test. Guard the whole test with #ifdef BENCHMARK (matching the existing benchmark in this file) so it stays out of normal test runs; dup_faithful_copy continues to cover correctness of the switch_event_dup change.

Signed-off-by: Calvin Ellison <cellison@youmail.com>
…t loops

Address review feedback on the switch_event_dup change:

- src/switch_event.c: the fast-path comment claimed todup's headers are "already unique" unconditionally. That only holds when EF_UNIQ_HEADERS is set (names are deduped on insert); reword to scope the claim to that case and to the verbatim-copy intent.

- tests/unit/switch_event.c: expand one-line loops to braced form per the SignalWire coding guidelines.

Signed-off-by: Calvin Ellison <cellison@youmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants