You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/learning-paths/servers-and-cloud-computing/microbenchmark-network-iperf3/basic-microbenchmarking.md
+16-12Lines changed: 16 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,28 +1,31 @@
1
1
---
2
-
title: Microbenchmark Existing Network Connection
2
+
title: Microbenchmark the network connection
3
3
weight: 3
4
4
5
5
### FIXED, DO NOT MODIFY
6
6
layout: learningpathall
7
7
---
8
8
9
-
## Microbenchmark TCP Connection
9
+
## Microbenchmark the TCP connection
10
10
11
+
You can microbenchmark the bandwidth between the client and server.
11
12
12
-
First we will microbenchmark the bandwidth between the client and server. First start the `iperf` server on the server node with the following command.
13
+
First, start `iperf`in server mode on the server system with the following command:
13
14
14
15
```bash
15
16
iperf3 -s
16
17
```
17
18
19
+
You see the output, indicating the server is ready:
By default, the server listens on port 5201. Use the `-p` flag to specify another port if it is in use.
27
+
28
+
The default server port is 5201. Use the `-p` flag to specify another port if it is in use.
26
29
27
30
{{% notice Tip %}}
28
31
If you already have an `iperf3` server running, you can manually kill the process with the following command.
@@ -31,13 +34,14 @@ If you already have an `iperf3` server running, you can manually kill the proces
31
34
```
32
35
{{% /notice %}}
33
36
34
-
35
37
Next, on the client node, run the following command to run a simple 10-second microbenchmark using the TCP protocol.
36
38
37
39
```bash
38
40
iperf3 -c SERVER -V
39
41
```
40
42
43
+
The output is similar to:
44
+
41
45
```output
42
46
...
43
47
[ 5] local 10.248.213.97 port 42176 connected to 10.248.213.104 port 5201
@@ -65,27 +69,27 @@ rcv_tcp_congestion cubic
65
69
iperf Done.
66
70
```
67
71
68
-
- The`Cwnd`stands for the control window size and corresponds to the allowed number of TCP transactions inflight before receiving an acknowledgment `ACK` from the server. This adjusts dynamically to not overwhelm the receiver and adjust for variable link connection strengths.
72
+
- The`Cwnd`column prints the control window size and corresponds to the allowed number of TCP transactions inflight before receiving an acknowledgment `ACK` from the server. This adjusts dynamically to not overwhelm the receiver and adjust for variable link connection strengths.
69
73
70
-
- The `CPU Utilization` row shows both the usage on the sender and receiver. If you are migrating your workload to a different platform, such as from `x86` to `AArch64`, there may be subtle variations.
74
+
- The `CPU Utilization` row shows both the usage on the sender and receiver. If you are migrating your workload to a different platform, such as from x86 to Arm, there may be variations.
71
75
72
76
- The `snd_tcp_congestion cubic` abd `rcv_tcp_congestion cubic` variables show the congestion control algorithm used.
73
77
74
-
- This `bitrate` shows the throughput achieved under this microbenchmark. As we can see from the above, we have saturated the 5 Gbps bandwidth available to our`t4g.xlarge` AWS instance.
78
+
- This `bitrate` shows the throughput achieved under this microbenchmark. As you can see, the 5 Gbps bandwidth available to the`t4g.xlarge` AWS instance is saturated.
We can also microbenchmark the `UDP` protocol with the `-u` flag. As a reminder, UDP does not guarantee packet delivery with some packets being lost. As such we need to observe the statistics on the server side to see the % of packet lost and the variation in packet arrival time (jitter). The UDP protocol is widely used in applications that need timely packet delivery, such as online gaming on video calls.
84
+
You can also microbenchmark the `UDP` protocol with the `-u` flag. As a reminder, UDP does not guarantee packet delivery with some packets being lost. As such you need to observe the statistics on the server side to see the percent of packets lost and the variation in packet arrival time (jitter). The UDP protocol is widely used in applications that need timely packet delivery, such as online gaming and video calls.
81
85
82
86
Run the following command from the client to send 2 parallel UDP streams with the `-P 2` option.
83
87
84
88
```bash
85
89
iperf3 -c SERVER -V -u -P 2
86
90
```
87
91
88
-
Looking at the server output we can observe 0% of packets where lost for our short test.
92
+
Looking at the server output you observe 0% of packets where lost for the short test.
89
93
90
94
```output
91
95
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
@@ -94,7 +98,7 @@ Looking at the server output we can observe 0% of packets where lost for our sho
For this demonstration I will be using instances available from AWS within a virtual private cloud (VPC)
11
+
To perform network performance testing you need two Linux computers. You can use AWS EC2 instances with Graviton processors or any other Linux virtual machines from another cloud service provider.
12
12
13
-
Create 2 Arm-based linux instances, 1 to act as the server and the other to act as the client. In this tutorial I will be using two `t4g.xlarge` instance running Ubuntu 22.04 LTS.
13
+
You will also experiment with a local computer and a cloud instance to learn the networking performance differences compared to two cloud instances.
14
14
15
+
The instructions below use EC2 instances from AWS connected in a virtual private cloud (VPC).
15
16
16
-
### Install dependencies
17
+
To get started, create two Arm-based Linux instances, one system to act as the server and the other to act as the client. The instructions below use two `t4g.xlarge` instances running Ubuntu 24.04 LTS.
17
18
18
-
Run the following command to install the microbenchmark tool, `iperf3`.
19
+
### Install software dependencies
20
+
21
+
Use the commands below to install `iperf3`, a powerful and flexible open-source command-line tool used for network performance measurement and tuning. It allows network administrators and engineers to actively measure the maximum achievable bandwidth on IP networks.
22
+
23
+
Run the following on both systems:
19
24
20
25
```bash
21
26
sudo apt update
22
27
sudo apt install iperf3 -y
23
28
```
24
29
30
+
{{% notice Note %}}
31
+
If you are prompted to start `iperf3` as a daemon you can answer no.
32
+
{{% /notice %}}
25
33
26
-
###Update Security Rules
34
+
## Update Security Rules
27
35
28
-
Next, we need to update the default security rules to enable specific inbound and outbound protocols. From the AWS console, navigate to the security tab. Edit the inbound rules to enable `ICMP`, `UDP` and `TCP` traffic to enable communication between the client and server
36
+
If you are working in a cloud environment like AWS, you need to update the default security rules to enable specific inbound and outbound protocols.
29
37
38
+
From the AWS console, navigate to the security tab. Edit the inbound rules to enable `ICMP`, `UDP` and `TCP` traffic to enable communication between the client and server systems.
30
39
31
40
32
41
33
42
{{% notice Note %}}
34
-
For security set the source and port ranges to those that are being used
43
+
For additional security set the source and port ranges to the values being used. A good solution is to open TCP port 5201 and all UDP ports and use your security group as the source. This doesn't open any traffic from outside AWS.
35
44
{{% /notice %}}
36
45
46
+
## Update the local DNS
37
47
38
-
### Update local DNS
48
+
To avoid using IP addresses directly, add the IP address of the other system to the `/etc/hosts` file.
39
49
40
-
For readability, we will add the server IP address and an alias to the local DNS cache in `/etc/hosts`. The local IP address of the server and client can be found in the AWS dashboard.
50
+
The local IP address of the server and client can be found in the AWS dashboard. You can also use commands like `ifconfig`, `hostname -I`, or `ip address` to find your local IP address.
41
51
42
-
On the client, add the IP address of the server to the `/etc/hosts` file. Likewise on the server add the IP address of the client to the `/etc/hosts` file.
52
+
On the client, add the IP address of the server to the `/etc/hosts` file with name `SERVER`.
43
53
44
-
.
54
+
```output
55
+
127.0.0.1 localhost
56
+
10.248.213.104 SERVER
57
+
```
45
58
46
-
### Confirm server is reachable
59
+
Repeat the same thing on the server and add the IP address of the client to the `/etc/hosts` file with the name `CLIENT`.
47
60
48
-
Finally, confirm the client can reach the server with the ping command below. As a reference we also ping the localhost.
61
+
## Confirm server is reachable
62
+
63
+
Finally, confirm the client can reach the server with the ping command below. As a reference you can also ping the localhost.
49
64
50
65
```bash
51
66
ping SERVER -c 3 && ping 127.0.0.1 -c 3
52
67
```
53
68
54
-
The output below shows that both SERVER and localhost (127.0.0.1) are reachable. Naturally, on this system local host response tile is ~10x faster than the server. Your results will vary depending on geographic colocation and other networking factors.
69
+
The output below shows that both SERVER and localhost (127.0.0.1) are reachable. Naturally, the local host response time is ~10x faster than the server. Your results will vary depending on geographic location of the systems and other networking factors.
55
70
56
71
```output
57
72
PING SERVER (10.248.213.104) 56(84) bytes of data.
Copy file name to clipboardExpand all lines: content/learning-paths/servers-and-cloud-computing/microbenchmark-network-iperf3/simulating-network-conditions.md
+23-9Lines changed: 23 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,14 +1,16 @@
1
1
---
2
-
title: Simulating Different Network Conditions
2
+
title: Simulating different network conditions
3
3
weight: 4
4
4
5
5
### FIXED, DO NOT MODIFY
6
6
layout: learningpathall
7
7
---
8
8
9
-
## Adding a delay to a TCP connection
9
+
## Add a delay to the TCP connection
10
10
11
-
The linux `tc` utility can be used to manipulate traffic control settings. First, find the name of interface with the following command.
11
+
The Linux `tc` utility can be used to manipulate traffic control settings.
12
+
13
+
First, on the client system, find the name of network interface with the following command:
12
14
13
15
```bash
14
16
ip addr show
@@ -32,14 +34,21 @@ The output below shows the `ens5` network interface device (NIC) is the device w
32
34
33
35
```
34
36
35
-
Run the following command to add an emulated delay of 10ms on `ens5`.
37
+
Run the following command on the client system to add an emulated delay of 10ms on `ens5`.
36
38
37
39
```bash
38
40
sudo tc qdisc add dev ens5 root netem delay 10ms
39
41
```
40
42
41
-
Rerunning the basic TCP test (`iperf3 -c SERVER -V`) with a delay we observe the `Cwnd` size has grew larger to compensate for the longer response time. Additionally, the bitrate has dropped from ~4.9 to ~2.3 `Gbit/sec`.
43
+
Rerun the basic TCP test as before on the client:
44
+
45
+
```bash
46
+
iperf3 -c SERVER -V
47
+
```
42
48
49
+
Observe that the `Cwnd` size has grew larger to compensate for the longer response time.
50
+
51
+
Additionally, the bitrate has dropped from ~4.9 to ~2.3 `Gbit/sec`.
43
52
44
53
```output
45
54
[ 5] local 10.248.213.97 port 43170 connected to 10.248.213.104 port 5201
@@ -67,25 +76,30 @@ rcv_tcp_congestion cubic
67
76
iperf Done.
68
77
```
69
78
70
-
### Simulating Packet Loss
79
+
### Simulate packet loss
80
+
81
+
To test the resiliency of a distributed application you can add a simulated packet loss of 1%. As opposed to a 10ms delay, this will result in no acknowledgment being received for 1% of packets. Given TCP is a lossless protocol a retry must be sent.
71
82
72
-
To test the resiliency of a distributed application we can add a simulated packet loss of 1%. As opposed to a 10ms delay, this will result in no acknowledgment being received for 1% of packets. Given TCP is a lossless protocol a retry must be sent.
83
+
Run these commands on the client system:
73
84
74
85
```bash
75
86
sudo tc qdisc del dev ens5 root
76
87
sudo tc qdisc add dev ens5 root netem loss 1%
77
88
```
78
89
79
-
Rerunning the basic TCP test we observe an increased number of retries (`Retr`) and a corresponding drop in bitrate.
90
+
Rerunning the basic TCP test you see an increased number of retries (`Retr`) and a corresponding drop in bitrate.
Please see the `tc`[user documentation](https://man7.org/linux/man-pages/man8/tc.8.html) for the different ways to simulate different perturbation and your systems resiliency to such events.
105
+
Refer to the `tc`[user documentation](https://man7.org/linux/man-pages/man8/tc.8.html) for the different ways to simulate perturbation and check resiliency.
Now we can observe ways to mitigate performance degradation due to events such as packet loss. In this example, I will connect to the AWS server node from my local machine to demonstrate a longer response time. Please check the `iperf3` installation guide on the [official documentation](https://iperf.fr/iperf-download.php) if you're not using Ubuntu. As the output below shows we have a larger round trip time in excess of 40ms.
11
+
You can look at ways to mitigate performance degradation due to events such as packet loss.
12
12
13
-
```output
14
-
5 packets transmitted, 5 packets received, 0.0% packet loss
15
-
round-trip min/avg/max/stddev = 44.896/46.967/49.279/1.444 ms
13
+
In this example, you will connect to the server node a local machine to demonstrate a longer response time. Check the `iperf3`[installation guide](https://iperf.fr/iperf-download.php) to install `iperf3` on other operating systems.
14
+
15
+
Make sure to set the server security group to accept the TCP connection from your local computer IP address. You will also need to use the public IP for the cloud instance.
16
+
17
+
Running `iperf3` on the local machine and connecting to the cloud server shows a longer round trip time, in this example more than 40ms.
18
+
19
+
On your local computer run:
20
+
21
+
```bash
22
+
iperf3 -c <server-public-IP> -V
16
23
```
17
24
18
-
Running a standard TCP client connection with the `iperf3 -c SERVER -V` command shows an average bitrate of 157 Mbps.
25
+
Running a standard TCP client connection with `iperf3`shows an average bitrate of 157 Mbps compared to over 2 Gbps when the client and server are both in AWS.
19
26
20
27
```output
21
28
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
@@ -26,13 +33,11 @@ Test Complete. Summary Results:
On the server, we can configure linux kernel runtime parameters with the `sysctl` command.
38
+
On the server, your can configure Linux kernel runtime parameters with the `sysctl` command.
34
39
35
-
There are a plenthora of dials to tune that relate to performance and security. The following command can be used to list all available dials. The corresponding [kernel documentation](https://docs.kernel.org/networking/ip-sysctl.html#ip-sysctl)can provide a more detailed description of each parameter.
40
+
There are a plenthora of values to tune that relate to performance and security. The following command can be used to list all available options. The [Linux kernel documentation](https://docs.kernel.org/networking/ip-sysctl.html#ip-sysctl)provides a more detailed description of each parameter.
You can increase the read and write max buffer sizes of the kernel on the server to enable more data to be held. This tradeoff results in increased memory utilization.
48
54
49
-
We can increase the read and write max buffer sizes of the kernel on the server to enable more data to be held. This is at the tradeoff of increased memory utilisation. run the following commands from the server.
55
+
To try it, run the following commands on the server:
Restart the `iperf3` server. Run the `iperf3 -c SERVER -V` command from the client leads to significantly improved bitrate with no modification on the client side.
62
+
Restart the `iperf3` server.
63
+
64
+
```bash
65
+
iperf3 -s
66
+
```
67
+
68
+
Run `iperf3` again on the local machine.
69
+
70
+
```bash
71
+
iperf3 -c <server-public-IP> -V
72
+
```
73
+
74
+
You see a significantly improved bitrate with no modification on the client side.
57
75
58
76
```output
59
77
Test Complete. Summary Results:
@@ -63,4 +81,4 @@ Test Complete. Summary Results:
63
81
64
82
```
65
83
66
-
This learning path serves as an introduction to microbenchmarking and performance tuning. Which parameters to adjust depends on your own use case and non-functional performance requirements of your system.
84
+
You now have an introduction to networking microbenchmarking and performance tuning.
0 commit comments