Enhance iperf3 to run for multiple sessions#551
Conversation
| { | ||
| threads_to_iterate=1 | ||
| nr_node=$(echo /sys/devices/system/node/node* | wc -w) | ||
| threads_to_iterate=1 |
There was a problem hiding this comment.
thanks, any consideration to move these variables out of the setup_threads_to_iterate()?
There was a problem hiding this comment.
My intention was to expose the value of cores_per_node which was local to setup_threads_to_iterate(). But thanks to you, since you pointed this out, I was thinking maybe I can call keep all the variables inside the setup_threads_to_iterate(), export cores_per_node and call this function once in the run script to get and use the value. That way the implementation seems to be cleaner and safer I guess.
| runtime: | ||
| protocol: | ||
| sessions: | ||
| port: |
There was a problem hiding this comment.
is port necessary to be part of the result root path?
There was a problem hiding this comment.
Yes you are right. It makes sense to remove -port from the result-root path. Will make the changes. Thanks much for the suggestion.
| val=$2 | ||
| shift 2 | ||
|
|
||
| [ -n "$key" ] && [ -n "$val" ] || continue |
There was a problem hiding this comment.
consider to use double brackets like [[, ]]
There was a problem hiding this comment.
Acknowledged. Thanks for catching that.
| tune_param() | ||
| { | ||
| set -- $tune | ||
| while [ $# -ge 2 ]; do |
There was a problem hiding this comment.
consider to use double brackets like [[, ]]
|
|
||
| build() | ||
| { | ||
| cd $srcdir/$pkgname |
There was a problem hiding this comment.
consider to use cd_src_pkg_dir by source $LKP_SRC/lib/tests/pkgbuild.sh
There was a problem hiding this comment.
Acknowledged. Thanks for the suggestion.
There was a problem hiding this comment.
Pull request overview
This PR enhances the iperf benchmark integration to support multi-session runs, additional runtime tuning options, and builds iperf from upstream sources for more consistent cross-distro behavior.
Changes:
- Add multi-session execution support with configurable
sessions/port, optionalnumactl, andsysctltuning inprograms/iperf/run. - Update iperf stats parsing to handle concatenated JSON from multiple sessions and report averaged throughput.
- Introduce a makepkg-based upstream
PKGBUILDand add new fixtures to unit-test iperf stats parsing for TCP/UDP single- and multi-session outputs.
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
programs/iperf/run |
Adds sessions/port/numactl/tune handling and multi-session orchestration. |
programs/iperf/parse |
Parses concatenated multi-session JSON and averages bps metrics. |
programs/iperf/meta.yaml |
Exposes new sessions, port, numactl, tune parameters. |
programs/iperf/pkg/PKGBUILD |
Adds upstream-source build packaging for iperf via makepkg. |
programs/iperf/pkg/depends |
Removes prior runtime dependency declaration. |
lib/sysinfo.sh |
Changes how NUMA/core topology vars are computed/exported. |
distro/adaptation-pkg/ubuntu |
Adds iperf to makepkg adaptation list. |
distro/adaptation-pkg/debian |
Adds iperf to makepkg adaptation list. |
spec/fixtures/stats/iperf/tcp |
Adds iperf3 TCP JSON fixture for stats invariance tests. |
spec/fixtures/stats/iperf/tcp.yaml |
Adds expected TCP stats output fixture. |
spec/fixtures/stats/iperf/tcp-multi-session |
Adds multi-session TCP concatenated JSON fixture. |
spec/fixtures/stats/iperf/tcp-multi-session.yaml |
Adds expected multi-session TCP stats output fixture. |
spec/fixtures/stats/iperf/udp |
Adds iperf3 UDP JSON fixture for stats invariance tests. |
spec/fixtures/stats/iperf/udp.yaml |
Adds expected UDP stats output fixture. |
spec/fixtures/stats/iperf/udp-multi-session |
Adds multi-session UDP concatenated JSON fixture. |
spec/fixtures/stats/iperf/udp-multi-session.yaml |
Adds expected multi-session UDP stats output fixture. |
Comments suppressed due to low confidence (1)
programs/iperf/pkg/depends:1
programs/iperf/run(andprograms/iperf-server/daemon) invokeiperf3, butprograms/iperf/pkg/dependsis now empty. On a clean rootfs this removes the only declared dependency that ensuresiperf3is present, and it can also causesbin/makepkg-deps/build_depends_pkgto fail with "empty deps for iperf". Add the appropriate dependency declaration (either restore a distro dependency likeiperf3, or list the makepkg package(s) that provideiperf3).
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| for i in `seq 1 $sessions`; do | ||
| if [ "$numactl" = true ]; then | ||
| numactl_server="numactl -C $(($cores_per_node + i))" | ||
| fi | ||
| $numactl_server iperf3 -s -B $server -p $(($port+i-1)) -1 > /dev/null & | ||
| done | ||
|
|
||
| for i in `seq 1 $sessions`; do | ||
| if [ "$numactl" = true ]; then | ||
| numactl_client="numactl -C $((0 + i))" | ||
| fi | ||
| $numactl_server iperf3 -c $server -p $(($port+i-1)) -i 10 -T $((0+i-1)) -J -t $runtime $opt_udp | ||
| done |
There was a problem hiding this comment.
The "multi-session" mode currently runs the client iperf3 commands sequentially (no &/wait). That means only one session is active at a time, which defeats the stated goal of running multiple client/server pairs concurrently to stress multiple cores. If true concurrency is required, start the client sessions in the background, capture each session’s JSON output to separate files, then wait and concatenate them in a deterministic order for parsing.
| for i in `seq 1 $sessions`; do | |
| if [ "$numactl" = true ]; then | |
| numactl_server="numactl -C $(($cores_per_node + i))" | |
| fi | |
| $numactl_server iperf3 -s -B $server -p $(($port+i-1)) -1 > /dev/null & | |
| done | |
| for i in `seq 1 $sessions`; do | |
| if [ "$numactl" = true ]; then | |
| numactl_client="numactl -C $((0 + i))" | |
| fi | |
| $numactl_server iperf3 -c $server -p $(($port+i-1)) -i 10 -T $((0+i-1)) -J -t $runtime $opt_udp | |
| done | |
| client_output_dir=$(mktemp -d) | |
| declare -a server_pids | |
| declare -a client_pids | |
| declare -a client_output_files | |
| client_cmd="" | |
| for i in `seq 1 $sessions`; do | |
| if [ "$numactl" = true ]; then | |
| numactl_server="numactl -C $(($cores_per_node + i))" | |
| else | |
| numactl_server="" | |
| fi | |
| $numactl_server iperf3 -s -B $server -p $(($port+i-1)) -1 > /dev/null & | |
| server_pids[$i]=$! | |
| done | |
| for i in `seq 1 $sessions`; do | |
| if [ "$numactl" = true ]; then | |
| numactl_client="numactl -C $((0 + i))" | |
| else | |
| numactl_client="" | |
| fi | |
| client_output_file="$client_output_dir/client-$i.json" | |
| client_output_files[$i]="$client_output_file" | |
| client_session_cmd="$numactl_client iperf3 -c $server -p $(($port+i-1)) -i 10 -T $((0+i-1)) -J -t $runtime $opt_udp" | |
| if [ -n "$client_cmd" ]; then | |
| client_cmd="${client_cmd}; " | |
| fi | |
| client_cmd="${client_cmd}${client_session_cmd}" | |
| eval "$client_session_cmd > \"$client_output_file\" &" | |
| client_pids[$i]=$! | |
| done | |
| for i in `seq 1 $sessions`; do | |
| wait "${client_pids[$i]}" | |
| done | |
| for i in `seq 1 $sessions`; do | |
| wait "${server_pids[$i]}" | |
| done | |
| for i in `seq 1 $sessions`; do | |
| cat "${client_output_files[$i]}" | |
| done | |
| rm -rf "$client_output_dir" |
| for i in `seq 1 $sessions`; do | ||
| if [ "$numactl" = true ]; then | ||
| numactl_server="numactl -C $(($cores_per_node + i))" | ||
| fi | ||
| $numactl_server iperf3 -s -B $server -p $(($port+i-1)) -1 > /dev/null & | ||
| done |
There was a problem hiding this comment.
Servers are started in the background and clients are started immediately afterward, but there is no readiness check to ensure each iperf3 -s is actually listening on its port before the corresponding client connects. This can cause intermittent connection failures (and leave iperf3 -s -1 processes lingering, waiting forever). Add a small per-port readiness loop (e.g., ss/lsof/nc) and/or a cleanup trap to kill spawned servers on failure.
| threads_to_iterate=1 | ||
| export nr_node=$(echo /sys/devices/system/node/node* | wc -w) | ||
|
|
||
| local threads_per_core | ||
| local cores_per_node | ||
| local cores_per_llc | ||
| local nr_l3 | ||
| local nr_l2 | ||
| threads_per_core=$(lscpu | awk '/Thread\(s\) per core:/ { print $4 }') | ||
| nr_l3=$(lscpu | grep 'L3' | awk -F '[()]' '{print $2}' | awk '{print $1}') | ||
| nr_l2=$(lscpu | grep 'L2' | awk -F '[()]' '{print $2}' | awk '{print $1}') | ||
| cores_per_llc=$((nr_l2 / nr_l3)) | ||
| cores_per_node=$((nr_cpu / nr_node / threads_per_core)) | ||
| threads_per_core=$(lscpu | awk '/Thread\(s\) per core:/ { print $4 }') | ||
| nr_l3=$(lscpu | grep 'L3' | awk -F '[()]' '{print $2}' | awk '{print $1}') | ||
| nr_l2=$(lscpu | grep 'L2' | awk -F '[()]' '{print $2}' | awk '{print $1}') | ||
| cores_per_llc=$((nr_l2 / nr_l3)) | ||
| export cores_per_node=$((nr_cpu / nr_node / threads_per_core)) |
There was a problem hiding this comment.
lib/sysinfo.sh now computes and exports cores_per_node at file load time using nr_cpu, but nr_cpu is not defined within this script. If the script is sourced before nr_cpu is exported by the harness, the arithmetic expansion will fail and can break callers. Consider moving this computation back inside setup_threads_to_iterate() (or guarding it with a check/default for nr_cpu).
| if [[ -n "$sessions" ]];then | ||
| for i in `seq 1 $sessions`; do | ||
| if [ "$numactl" = true ]; then | ||
| numactl_server="numactl -C $(($cores_per_node + i))" | ||
| fi | ||
| $numactl_server iperf3 -s -B $server -p $(($port+i-1)) -1 > /dev/null & | ||
| done | ||
|
|
||
| for i in `seq 1 $sessions`; do | ||
| if [ "$numactl" = true ]; then | ||
| numactl_client="numactl -C $((0 + i))" | ||
| fi | ||
| $numactl_server iperf3 -c $server -p $(($port+i-1)) -i 10 -T $((0+i-1)) -J -t $runtime $opt_udp |
There was a problem hiding this comment.
When sessions is set, this script starts iperf3 -s processes locally and then runs the clients against $server. If $server is a remote host (e.g. direct_server_ips is set), the local servers are unnecessary and -B $server will try to bind to an address not present on the client, causing failures. Consider restricting the local server spawning to localhost-only runs (or moving server spawning to the server-role program) and keeping the client-only behavior for remote server cases.
| if [[ -n "$sessions" ]];then | |
| for i in `seq 1 $sessions`; do | |
| if [ "$numactl" = true ]; then | |
| numactl_server="numactl -C $(($cores_per_node + i))" | |
| fi | |
| $numactl_server iperf3 -s -B $server -p $(($port+i-1)) -1 > /dev/null & | |
| done | |
| for i in `seq 1 $sessions`; do | |
| if [ "$numactl" = true ]; then | |
| numactl_client="numactl -C $((0 + i))" | |
| fi | |
| $numactl_server iperf3 -c $server -p $(($port+i-1)) -i 10 -T $((0+i-1)) -J -t $runtime $opt_udp | |
| server_is_localhost=false | |
| case "$server" in | |
| 127.0.0.1|::1|localhost) | |
| server_is_localhost=true | |
| ;; | |
| esac | |
| if [[ -n "$sessions" ]];then | |
| if [ "$server_is_localhost" = true ]; then | |
| for i in `seq 1 $sessions`; do | |
| if [ "$numactl" = true ]; then | |
| numactl_server="numactl -C $(($cores_per_node + i))" | |
| fi | |
| $numactl_server iperf3 -s -B $server -p $(($port+i-1)) -1 > /dev/null & | |
| done | |
| fi | |
| for i in `seq 1 $sessions`; do | |
| if [ "$numactl" = true ]; then | |
| numactl_client="numactl -C $((0 + i))" | |
| fi | |
| $numactl_client iperf3 -c $server -p $(($port+i-1)) -i 10 -T $((0+i-1)) -J -t $runtime $opt_udp |
| if [ "$numactl" = true ]; then | ||
| numactl_client="numactl -C $((0 + i))" | ||
| fi | ||
| $numactl_server iperf3 -c $server -p $(($port+i-1)) -i 10 -T $((0+i-1)) -J -t $runtime $opt_udp |
There was a problem hiding this comment.
In the multi-session client loop you build a numactl_client command, but the actual iperf3 invocation uses numactl_server again. This makes the client sessions run with the server CPU binding (and numactl_client is effectively unused). Switch the client invocation to use numactl_client (or a shared variable) so client/server bindings behave as intended.
| $numactl_server iperf3 -c $server -p $(($port+i-1)) -i 10 -T $((0+i-1)) -J -t $runtime $opt_udp | |
| $numactl_client iperf3 -c $server -p $(($port+i-1)) -i 10 -T $((0+i-1)) -J -t $runtime $opt_udp |
| client_cmd="iperf3 -t $runtime -J -c $server $opt_udp" | ||
| ${client_cmd} | ||
| fi | ||
|
|
||
| echo "$(date +'%F %T') ${client_cmd}" >> $RESULT_ROOT/reproduce.sh |
There was a problem hiding this comment.
client_cmd is only set in the single-session branch, but it is referenced unconditionally when appending to $RESULT_ROOT/reproduce.sh. When sessions is set this writes an empty/incorrect reproduce command. Define an appropriate reproduce command for the multi-session case (or gate the reproduce.sh write on client_cmd being set).
| make install | ||
| } | ||
|
|
||
| package() | ||
| { | ||
| benchmark_path="${pkgdir}/lkp/benchmarks/${pkgname}" | ||
| mkdir -p $benchmark_path | ||
| cp -r "$srcdir/$pkgname/" "$benchmark_path/". |
There was a problem hiding this comment.
This PKGBUILD runs make install during build(), which will install into the build host rather than staging into $pkgdir. For reproducible packaging (and to avoid polluting the build environment), keep build() to compilation only and perform installation in package() using DESTDIR="$pkgdir"/--prefix, packaging just the installed artifacts rather than copying the whole source tree.
| make install | |
| } | |
| package() | |
| { | |
| benchmark_path="${pkgdir}/lkp/benchmarks/${pkgname}" | |
| mkdir -p $benchmark_path | |
| cp -r "$srcdir/$pkgname/" "$benchmark_path/". | |
| } | |
| package() | |
| { | |
| cd $srcdir/$pkgname | |
| make DESTDIR="$pkgdir" install |
| pkgver=git | ||
| pkgrel=1 | ||
| url='https://github.com/esnet/iperf' | ||
| arch=('i386' 'x86_64' 'aarch64') |
There was a problem hiding this comment.
The arch list uses i386, but other repo PKGBUILDs use Arch-style i686 (e.g., programs/kernel-selftests/pkg/PKGBUILD:4). If this is built via makepkg, i386 may be treated as invalid/unsupported. Align the architecture identifiers with the existing PKGBUILD convention in this repo.
| arch=('i386' 'x86_64' 'aarch64') | |
| arch=('i686' 'x86_64' 'aarch64') |
| if [ "$numactl" = true ]; then | ||
| numactl_server="numactl -C $(($cores_per_node + i))" | ||
| fi | ||
| $numactl_server iperf3 -s -B $server -p $(($port+i-1)) -1 > /dev/null & |
There was a problem hiding this comment.
The new numactl handling here uses numactl -C ... (CPU affinity) only. That does not apply any NUMA memory policy (e.g., --membind/--interleave) and also bypasses the repo’s existing NUMA binding helpers (lib/numactl.sh, used by e.g. programs/memtier/run:26-83). If the intent is memory policy control, consider wiring this to parse_numa_node_binding/numa_node_binding so both CPU+memory bindings are handled consistently.
8def7b2 to
1a33c83
Compare
|
Thanks, @Suneeth-D could you help do a check of "Ruby / test-code (3.2) (pull_request)Failing after 1m"? |
1a33c83 to
0ba6c52
Compare
Get iperf3 microbenchmark package from a reliable source rather than using distro specific version of iperf which varies from distro to distro. Signed-off-by: Suneeth D <Suneeth.D@amd.com>
The patch has the following changes incorporated:- * Add sessions param to enable run multiple sessions of iperf3 server-client pairs. * In order to avoid server unavailability and port conflicts, add port param to run $sessions sessions of iperf3 server-client pairs on ports $port through $port+($sessions-1). * Modify the existing parser script, thereby enabling parse the "bps" for multiple iperf3 sessions Signed-off-by: Suneeth D <Suneeth.D@amd.com>
Add numactl param to enable iperf3 when run in multiple server-client pairs, with a specific memory placement policy. Signed-off-by: Suneeth D <Suneeth.D@amd.com>
Add tune param to run script and meta.yaml to pass the kernel parameters that needs to be tuned wrt networking before starting to run the iperf3 microbenchmark Signed-off-by: Suneeth D <Suneeth.D@amd.com>
Add appropiate yaml files and console logs for tcp/udp(single/multiple) sessions under spec/fixtures/stats dir for the iperf to have unit tests coverage for multiple sessions. Signed-off-by: Suneeth D <Suneeth.D@amd.com>
|
Thanks, some remaining issue to help check at |
0ba6c52 to
b504de3
Compare
|
thanks for the patch |
This patch series on the iperf3 microbenchmark package focuses on enhancing it to have few functionalities mentioned below and address the consequences of having those while not disturbing the existing iperf3 package on the latest intel/lkp-tests: master.
The enhancements are as follows:-
Modify PKGBUILD to have iperf3 installation from the upstream
repo as a single source rather than picking up distro specific which is
non-uniform among the distros.
Add sessions and port params to have multiple sessions of iperf3
running as client-server pairs which helps in identifying regression
on multi-core systems.
Add numactl param to enable running iperf3 in specified memory
policy on systems whers NR_NUMA_NODES > 1.
Add tune param to have certain kernel parameters pertaining to
network stack tuned before running the iperf3 microbenchmark.
Add unittests for iperf covering both single and multiple sessions.
Below is the yaml file I used for testing my patch series which has all the additional params that is introduced.
iperf3.yaml
#! jobs/iperf.yaml
suite: iperf
testcase: iperf
category: benchmark
runtime: 30s
sessions: 8
port: 25000
cluster: cs-localhost
if role client:
iperf:
protocol: tcp
numactl: true
tune:
- net.core.wmem_max 212992
- net.core.rmem_max 212992
- net.core.rmem_default 212992
- net.core.wmem_default 212992
- net.core.netdev_max_backlog 25000
- net.core.busy_poll 30
- net.core.busy_read 30
- net.core.netdev_budget 600
- net.core.rps_sock_flow_entries 65536
job_origin: jobs/iperf.yaml
arch: x86_64
node_roles: server client
UNIT TEST output after adding the patch series,
rake spec spec=stats
PWD = /home/suneeth/lkp-tests
ENV['LKP_SRC'] = /home/suneeth/lkp-tests
/usr/bin/ruby3.2 -I/var/lib/gems/3.2.0/gems/rspec-core-3.13.6/lib:/var/lib/gems/3.2.0/gems/rspec-support-3.13.7/lib /var/lib/gems/3.2.0/gems/rspec-core-3.13.6/exe/rspec --pattern spec/**{,/*/**}/stats_spec.rb
..........................................................................................................................................................................................................................................................................................................................................................................................F.....................................................
Failures:
stats scripts invariance: /home/suneeth/lkp-tests/spec/fixtures/stats/vmstat/1
Failure/Error: expect(new_stat).to eq old_stat.chomp
expected: "time: 1420978715\nprocs.r: 2\nprocs.b: 0\nmemory.swpd: 0\nmemory.free: 224636\nmemory.buff: 33636\nm...\nio.bo: 4\nsystem.in: 753\nsystem.cs: 2187\ncpu.us: 2\ncpu.sy: 1\ncpu.id: 98\ncpu.wa: 0\ncpu.st: 0"
got: "time: 1421007515\nprocs.r: 2\nprocs.b: 0\nmemory.swpd: 0\nmemory.free: 224636\nmemory.buff: 33636\nm...\nio.bo: 4\nsystem.in: 753\nsystem.cs: 2187\ncpu.us: 2\ncpu.sy: 1\ncpu.id: 98\ncpu.wa: 0\ncpu.st: 0"
(compared using ==)
Diff:
@@ -1,4 +1,4 @@
-time: 1420978715
+time: 1421007515
procs.r: 2
procs.b: 0
memory.swpd: 0
@@ -16,7 +16,7 @@
cpu.id: 98
cpu.wa: 0
cpu.st: 0
-time: 1420978716
+time: 1421007516
procs.r: 0
procs.b: 0
memory.swpd: 0
./spec/unit/lib/stats_spec.rb:26:in `block (4 levels) in <top (required)>'
Finished in 1 minute 58.13 seconds (files took 0.33267 seconds to load)
432 examples, 1 failure
Failed examples:
rspec ./spec/unit/lib/stats_spec.rb[1:1:379] # stats scripts invariance: /home/suneeth/lkp-tests/spec/fixtures/stats/vmstat/1
/usr/bin/ruby3.2 -I/var/lib/gems/3.2.0/gems/rspec-core-3.13.6/lib:/var/lib/gems/3.2.0/gems/rspec-support-3.13.7/lib /var/lib/gems/3.2.0/gems/rspec-core-3.13.6/exe/rspec --pattern spec/**{,/*/**}/stats_spec.rb failed
NOTE: The failure on vmstat unit testcase is already there on latest master (f182b61)