Skip to content

Commit ec4e99e

Browse files
author
Your Name
committed
initial commit
1 parent 6f14d6a commit ec4e99e

11 files changed

Lines changed: 325 additions & 49 deletions

File tree

‎content/learning-paths/servers-and-cloud-computing/microbenchmark-network-iperf3/Setup.md‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
---
2+
title: Setup
3+
weight: 2
4+
5+
### FIXED, DO NOT MODIFY
6+
layout: learningpathall
7+
---
8+
9+
### Setup
10+
11+
For this demonstration I will be using instances available from AWS within a virtual private cloud (VPC)
12+
13+
Create 2 Arm-based linux instances, 1 to act as the server and the other to act as the client. In this tutorial I will be using an `t4g.xlarge` instance running Ubuntu 22.04 LTS.
14+
15+
16+
### Install dependencies
17+
18+
Run the following command to install the microbenchmark tool, `iperf3`.
19+
20+
```bash
21+
sudo apt update
22+
sudo apt install iperf3 -y
23+
```
24+
25+
26+
### Update Security Rules
27+
28+
Next, we need to update the default security rules to enable specific inbound and outbound protocols. From the AWS console, navigate to the security tab. Edit the inbound rules to enable `ICMP`, `UDP` and `TCP` traffic. Please note: for security we recommend updating the source to be the IP addresses of the client and server respectively.
29+
30+
![example_traffic](./example_traffic_rules.png)
31+
32+
33+
### Update local DNS
34+
35+
For readability, we will add the server IP address and an alias name to the local DNS cache in `/etc/hosts`.
36+
37+
On the client, add the IP address of the server to the `/etc/hosts` file. Likewise on the server add the IP address of the client to the `/etc/hosts` file.
38+
39+
![server-ip](./server-ip.png).
40+
41+
### Confirm server is reachable
42+
43+
Finally, confirm the client can reach the server with the ping command below.
44+
45+
```bash
46+
ping SERVER -c 3 && ping 127.0.0.1 -c 3
47+
```
48+
49+
The output below shows that both SERVER and localhost (127.0.0.1) are reachable. Naturally, on this system local host response tile is ~10x faster than the server. Your results will vary depending on geographic colocation and other networking factors.
50+
51+
```output
52+
PING SERVER (10.248.213.104) 56(84) bytes of data.
53+
64 bytes from SERVER (10.248.213.104): icmp_seq=1 ttl=64 time=0.217 ms
54+
64 bytes from SERVER (10.248.213.104): icmp_seq=2 ttl=64 time=0.218 ms
55+
64 bytes from SERVER (10.248.213.104): icmp_seq=3 ttl=64 time=0.219 ms
56+
57+
--- SERVER ping statistics ---
58+
3 packets transmitted, 3 received, 0% packet loss, time 2056ms
59+
rtt min/avg/max/mdev = 0.217/0.218/0.219/0.000 ms
60+
PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
61+
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.022 ms
62+
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.032 ms
63+
64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.029 ms
64+
65+
--- 127.0.0.1 ping statistics ---
66+
3 packets transmitted, 3 received, 0% packet loss, time 2046ms
67+
rtt min/avg/max/mdev = 0.022/0.027/0.032/0.004 ms
68+
69+
```
70+

‎content/learning-paths/servers-and-cloud-computing/microbenchmark-network-iperf3/_index.md‎

Lines changed: 16 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -1,48 +1,38 @@
11
---
2-
title: PLACEHOLDER TITLE
2+
title: Get started with network microbenchmarking and tuning with iperf3
33

4-
minutes_to_complete: 10
4+
minutes_to_complete: 30
55

6-
who_is_this_for: PLACEHOLDER SENTENCE
6+
who_is_this_for: Network engineers, sys admins or application developers
77

88
learning_objectives:
9-
- PLACEHOLDER OBJECTIVE 1
10-
- PLACEHOLDER OBJECTIVE 2
9+
- Understand how to use the iperf3 tool to microbenchmark different network conditions
10+
- Understand how to use the tc tool to simulate different network environments
11+
- Understand basic runtime parameters to tune performance for your application
1112

1213
prerequisites:
13-
- PLACEHOLDER PREREQ 1
14-
- PLACEHOLDER PREREQ 2
14+
- Foundational understanding on networking principles such as TCP/IP and UDP.
15+
- Access to change inbound and outbound security rules or access to physical hardware
1516

16-
author: PLACEHOLDER NAME
17+
author: Kieran Hejmadi
1718

1819
### Tags
19-
skilllevels: PLACEHOLDER SKILLLEVEL
20-
subjects: PLACEHOLDER SUBJECT
20+
skilllevels: Introductory
21+
subjects: Performance and Optimization
2122
armips:
22-
- PLACEHOLDER IP A
23-
- PLACEHOLDER IP B
23+
- Neoverse
2424
tools_software_languages:
25-
- PLACEHOLDER TOOL OR SOFTWARE C
26-
- PLACEHOLDER TOOL OR SOFTWARE D
25+
- iperf3
2726
operatingsystems:
28-
- PLACEHOLDER OS G
27+
- Linux
2928

3029

3130

3231
further_reading:
3332
- resource:
34-
title: PLACEHOLDER MANUAL
35-
link: PLACEHOLDER MANUAL LINK
33+
title: iperf3 user manual
34+
link: https://iperf.fr/iperf-doc.php
3635
type: documentation
37-
- resource:
38-
title: PLACEHOLDER BLOG
39-
link: PLACEHOLDER BLOG LINK
40-
type: blog
41-
- resource:
42-
title: PLACEHOLDER GENERAL WEBSITE
43-
link: PLACEHOLDER GENERAL WEBSITE LINK
44-
type: website
45-
4636

4737

4838
### FIXED, DO NOT MODIFY

‎content/learning-paths/servers-and-cloud-computing/microbenchmark-network-iperf3/basic-microbenchmarking.md‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
---
2+
title: Microbenchmark Existing Network Connection
3+
weight: 3
4+
5+
### FIXED, DO NOT MODIFY
6+
layout: learningpathall
7+
---
8+
9+
## Microbenchmark Existing Network Connection
10+
11+
12+
First we will microbenchmark the bandwidth between the client and server. First start the `iperf` server on the server node with the following command.
13+
14+
```bash
15+
iperf3 -s
16+
```
17+
18+
```output
19+
-----------------------------------------------------------
20+
Server listening on 5201 (test #1)
21+
-----------------------------------------------------------
22+
23+
24+
```
25+
By default, the server listens on port 5201. Use the `-p` flag to specify another port if it is in use.
26+
27+
Please note: If you're unable to start the `iperf` server, kill the process with the following command: `sudo kill $(pgrep iperf3)`
28+
29+
### Running Basic TCP Microbenchmark
30+
31+
Likewise, on the client node, run the following command to run a simple 10-second microbenchmark using the TCP protocol.
32+
33+
```bash
34+
iperf3 -c SERVER -V
35+
```
36+
37+
```output
38+
...
39+
[ 5] local 10.248.213.97 port 42176 connected to 10.248.213.104 port 5201
40+
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
41+
[ ID] Interval Transfer Bitrate Retr Cwnd
42+
[ 5] 0.00-1.00 sec 594 MBytes 4.98 Gbits/sec 0 1.48 MBytes
43+
[ 5] 1.00-2.00 sec 593 MBytes 4.97 Gbits/sec 0 2.07 MBytes
44+
[ 5] 2.00-3.00 sec 592 MBytes 4.97 Gbits/sec 0 2.07 MBytes
45+
[ 5] 3.00-4.00 sec 590 MBytes 4.96 Gbits/sec 0 2.07 MBytes
46+
[ 5] 4.00-5.00 sec 593 MBytes 4.97 Gbits/sec 0 2.18 MBytes
47+
[ 5] 5.00-6.00 sec 591 MBytes 4.96 Gbits/sec 0 2.18 MBytes
48+
[ 5] 6.00-7.00 sec 592 MBytes 4.97 Gbits/sec 0 2.18 MBytes
49+
[ 5] 7.00-8.00 sec 593 MBytes 4.97 Gbits/sec 0 2.18 MBytes
50+
[ 5] 8.00-9.00 sec 588 MBytes 4.93 Gbits/sec 0 2.18 MBytes
51+
[ 5] 9.00-10.00 sec 592 MBytes 4.96 Gbits/sec 0 2.18 MBytes
52+
- - - - - - - - - - - - - - - - - - - - - - - - -
53+
Test Complete. Summary Results:
54+
[ ID] Interval Transfer Bitrate Retr
55+
[ 5] 0.00-10.00 sec 5.78 GBytes 4.96 Gbits/sec 0 sender
56+
[ 5] 0.00-10.00 sec 5.78 GBytes 4.96 Gbits/sec receiver
57+
CPU Utilization: local/sender 5.3% (0.1%u/5.2%s), remote/receiver 26.7% (1.2%u/25.5%s)
58+
snd_tcp_congestion cubic
59+
rcv_tcp_congestion cubic
60+
61+
iperf Done.
62+
```
63+
64+
The important information to observe is `Cwnd`, this stands for the control window size and corresponds to the allowed number of TCP transactions inflight before receiving an acknowledgment `ACK` from the server. This adjusts dynamically to not overwhelm the receiver and adjust for variable link connection strengths.
65+
66+
The `CPU Utilization` row shows both the usage on the sender and receiver. If you are migrating your workload to a different platform, such as from `x86` to `AArch64`, there may be subtle variations.
67+
68+
Finally the `snd_tcp_congestion cubic` abd `rcv_tcp_congestion cubic` variables show the congestion control algorithm used.
69+
70+
This test has saturated the 5 Gbps bandwidth available to our `t4g.xlarge` AWS instance.
71+
72+
![instance-network-size](./instance-network-size.png)
73+
74+
### Running Basic UDP Microbenchmark
75+
76+
We can also microbenchmark the `UDP` protocol with the `-u` flag. As a reminder, UDP does not guarantee packat delivery with some packets being lost. As such we need to observe the statistics on the server side to see the % of packet lost and the variation in packet arrival time (jitter). The UDP protocol is widely used in applications that need timely packet delivery, such as online gaming on video calls.
77+
78+
Run the following command from the client to send 2 parallel UDP streams with the `-P 2` option.
79+
80+
```bash
81+
iperf3 -c SERVER -V -u -P 2
82+
```
83+
84+
Looking at the server output we can observe 0% of packets where lost for our short test.
85+
86+
```output
87+
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
88+
[ 5] 0.00-10.00 sec 1.25 MBytes 1.05 Mbits/sec 0.016 ms 0/147 (0%) receiver
89+
[ 6] 0.00-10.00 sec 1.25 MBytes 1.05 Mbits/sec 0.014 ms 0/147 (0%) receiver
90+
[SUM] 0.00-10.00 sec 2.51 MBytes 2.10 Mbits/sec 0.015 ms 0/294 (0%) receiver
91+
```
92+
93+
Additionally on the client side, our 2 streams maxed out 2 of our 4 core system.
94+
95+
```output
96+
CPU Utilization: local/sender 200.3% (200.3%u/0.0%s), remote/receiver 0.2% (0.0%u/0.2%s)
97+
```

‎content/learning-paths/servers-and-cloud-computing/microbenchmark-network-iperf3/example-picture.png‎

-61.7 KB
Binary file not shown.

‎content/learning-paths/servers-and-cloud-computing/microbenchmark-network-iperf3/example_traffic_rules.png‎

29 KB
Loading

‎content/learning-paths/servers-and-cloud-computing/microbenchmark-network-iperf3/how-to-1.md‎

Lines changed: 0 additions & 10 deletions
This file was deleted.

‎content/learning-paths/servers-and-cloud-computing/microbenchmark-network-iperf3/how-to-2.md‎

Lines changed: 0 additions & 13 deletions
This file was deleted.

‎content/learning-paths/servers-and-cloud-computing/microbenchmark-network-iperf3/instance-network-size.png‎

99.8 KB
Loading

‎content/learning-paths/servers-and-cloud-computing/microbenchmark-network-iperf3/server-ip.png‎

14.8 KB
Loading

‎content/learning-paths/servers-and-cloud-computing/microbenchmark-network-iperf3/simulating-network-conditions.md‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
---
2+
title: Simulating Different Scenarios
3+
weight: 4
4+
5+
### FIXED, DO NOT MODIFY
6+
layout: learningpathall
7+
---
8+
9+
## Adding a delay to a TCP connection
10+
11+
The linux `tc` utility can be used to manipulate traffic control settings. First, find the name of connections with the following command.
12+
13+
```bash
14+
ip addr show
15+
```
16+
17+
The output below shows the `ens5` network interface device (NIC) is the device we want to manipulate.
18+
19+
```output
20+
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
21+
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
22+
inet 127.0.0.1/8 scope host lo
23+
valid_lft forever preferred_lft forever
24+
inet6 ::1/128 scope host noprefixroute
25+
valid_lft forever preferred_lft forever
26+
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
27+
link/ether 0a:92:1b:a9:63:29 brd ff:ff:ff:ff:ff:ff
28+
inet 10.248.213.97/26 metric 100 brd 10.248.213.127 scope global dynamic ens5
29+
valid_lft 1984sec preferred_lft 1984sec
30+
inet6 fe80::892:1bff:fea9:6329/64 scope link
31+
valid_lft forever preferred_lft forever
32+
33+
```
34+
35+
Run the following command to add an emulated delay of 10ms on `ens5`.
36+
37+
```bash
38+
sudo tc qdisc add dev ens5 root netem delay 10ms
39+
```
40+
41+
Rerunning the basic TCP test with a delay we observe the `Cwnd` size has grew larger to compensate for the longer response time. Additionally, the bitrate has dropped from ~4.9 to ~2.3 `Gbit/sec`.
42+
43+
44+
```output
45+
[ 5] local 10.248.213.97 port 43170 connected to 10.248.213.104 port 5201
46+
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
47+
[ ID] Interval Transfer Bitrate Retr Cwnd
48+
[ 5] 0.00-1.00 sec 282 MBytes 2.36 Gbits/sec 0 8.02 MBytes
49+
[ 5] 1.00-2.00 sec 302 MBytes 2.53 Gbits/sec 0 8.02 MBytes
50+
[ 5] 2.00-3.00 sec 301 MBytes 2.52 Gbits/sec 0 8.02 MBytes
51+
[ 5] 3.00-4.00 sec 302 MBytes 2.54 Gbits/sec 0 8.02 MBytes
52+
[ 5] 4.00-5.00 sec 302 MBytes 2.53 Gbits/sec 0 8.02 MBytes
53+
[ 5] 5.00-6.00 sec 304 MBytes 2.55 Gbits/sec 0 8.02 MBytes
54+
[ 5] 6.00-7.00 sec 302 MBytes 2.53 Gbits/sec 0 8.02 MBytes
55+
[ 5] 7.00-8.00 sec 303 MBytes 2.54 Gbits/sec 0 8.02 MBytes
56+
[ 5] 8.00-9.00 sec 303 MBytes 2.54 Gbits/sec 0 8.02 MBytes
57+
[ 5] 9.00-10.00 sec 301 MBytes 2.53 Gbits/sec 0 8.02 MBytes
58+
- - - - - - - - - - - - - - - - - - - - - - - - -
59+
Test Complete. Summary Results:
60+
[ ID] Interval Transfer Bitrate Retr
61+
[ 5] 0.00-10.00 sec 2.93 GBytes 2.52 Gbits/sec 0 sender
62+
[ 5] 0.00-10.01 sec 2.93 GBytes 2.52 Gbits/sec receiver
63+
CPU Utilization: local/sender 3.4% (0.0%u/3.4%s), remote/receiver 11.0% (0.4%u/10.7%s)
64+
snd_tcp_congestion cubic
65+
rcv_tcp_congestion cubic
66+
67+
iperf Done.
68+
```
69+
70+
### Simulating Packet Loss
71+
72+
To test the resiliency of a distributed application we can add a simulated packet loss of 1%. As opposed to a 10ms delay, this will result in no acknowledgment being received for 1% of packets. Given TCP is a lossless protocol a retry must be sent.
73+
74+
```bash
75+
sudo tc qdisc del dev ens5 root
76+
sudo tc qdisc add dev ens5 root netem loss 1%
77+
```
78+
79+
Rerunning the basic TCP test we no observe a significant number of retries (`Retr`) and a corresponding drop in bitrate.
80+
81+
```bash
82+
iperf3 -c SERVER -V
83+
```
84+
```output
85+
Test Complete. Summary Results:
86+
[ ID] Interval Transfer Bitrate Retr
87+
[ 5] 0.00-10.00 sec 4.41 GBytes 3.78 Gbits/sec 5030 sender
88+
[ 5] 0.00-10.00 sec 4.40 GBytes 3.78 Gbits/sec receiver
89+
```
90+

0 commit comments

Comments
 (0)