|
6 | 6 |
|
7 | 7 | The objective of the network service is to make several instances communicate with each other as if they were all connected to a single private network, no matter if they are on the same machine or not. |
8 | 8 |
|
| 9 | +It also brings a network isolation layer between instances. |
| 10 | + |
9 | 11 | ### Suggested approach |
10 | 12 |
|
11 | | -To try to solve this problem, the network service is responsible for creating, configuring and deleting virtual network interfaces used to connect instances to each other within a cluster. |
| 13 | +To try to solve this problem, the agent service has to use a network library to setup network interfaces, namespaces, routing tables and iptables rules for each new node or new instance. |
12 | 14 |
|
13 | | -This service has to be running on all nodes of the cluster. |
| 15 | +Each instance run in its own network namespace. Its default network interface is a veth one, paired with another veth interface outside the instance namespace. |
14 | 16 |
|
15 | | -When adding a new node to a cluster, a call to the network service must be done to set up a new container network interface (CNI) service. There is one CNI service per node, and that service bridges the networking interfaces associated with the instances running on the node it controls to the rest of the cluster. |
| 17 | +All instance's veth interfaces (outside their instance's namespace) running on the same node are connected to a single bridge interface called Container Network Interface (CNI). |
16 | 18 |
|
17 | | -A call to the network service must also be made before creating a new instance in order to set up the virtual network interface connecting the instance to a CNI. |
| 19 | +The CNI is also connected to the default node's interface to allow communications from and to outside the node. |
18 | 20 |
|
19 | | -After the destruction of an instance, the network service must be called to remove the network interface associated with it and reconfigure the routing tables. |
| 21 | +Bellow is a representation of the network interfaces and namespaces in a Kudo node: |
20 | 22 |
|
21 | 23 |  |
22 | 24 |
|
23 | | -## API |
24 | | - |
25 | | -### Messages |
26 | | - |
27 | | -```protobuf |
28 | | -// Request structure used to create a new virtual network interface |
29 | | -message CreateNetworkInterfaceRequest { |
30 | | - string workload_id = 1; |
31 | | - string ip_address = 2; |
32 | | - repeated int32 ports = 3; |
33 | | -} |
34 | | -
|
35 | | -// Response structure returned after a new virtual interface has been created |
36 | | -message CreateNetworkInterfaceResponse { |
37 | | - string interface_name = 1; |
38 | | -} |
39 | | -
|
40 | | -// Request structure used to delete a virtual interface |
41 | | -message DeleteNetworkInterfaceRequest { |
42 | | - string workload_id = 1; |
43 | | -} |
44 | | -
|
45 | | -// Request structure used to setup a new node's network |
46 | | -message SetupRequest { |
47 | | - string ip_address = 1; |
48 | | - string sub_network = 2; |
49 | | -} |
50 | | -
|
51 | | -// Response structure returned after a new node's network has been setup |
52 | | -message SetupResponse { |
53 | | - string interface_name = 1; |
54 | | -} |
| 25 | +## API examples |
| 26 | + |
| 27 | +### Setup node |
| 28 | + |
| 29 | +Use `setup_node` function from `node` module each time you want to create a CNI and configure |
| 30 | +iptables rules for a new node. |
| 31 | + |
| 32 | +```rust |
| 33 | +let node_id = "node"; |
| 34 | +let node_ip_addr = Ipv4Addr::from_str("10.0.0.1").unwrap(); |
| 35 | +let node_ip_cidr = Ipv4Inet::new(node_ip_addr, 24).unwrap(); |
| 36 | + |
| 37 | +let request = SetupNodeRequest::new(node_id.to_string(), node_ip_cidr); |
| 38 | +let response = setup_node(request).unwrap(); |
| 39 | +println!("CNI name: {}", response.interface_name); |
| 40 | +``` |
| 41 | + |
| 42 | +After each node reboot, you need to reconfigure iptables running `setup_iptables` function from |
| 43 | +`node` module. |
| 44 | + |
| 45 | +```rust |
| 46 | +let node_id = "node"; |
| 47 | +let request = SetupIptablesRequest::new(node_id.to_string()); |
| 48 | + |
| 49 | +setup_iptables(request).unwrap(); |
| 50 | +``` |
| 51 | + |
| 52 | +### Setup instance |
| 53 | + |
| 54 | +Before running a new instance, please call `setup_instance` function from `instance` module to setup |
| 55 | +network namespace, interfaces and routing tables. |
| 56 | + |
| 57 | +```rust |
| 58 | +let node_id = "node"; |
| 59 | +let node_ip_addr = Ipv4Addr::from_str("10.0.0.1").unwrap(); |
| 60 | +let instance_id = "instance"; |
| 61 | +let instance_ip_addr = Ipv4Addr::from_str("10.0.0.2").unwrap(); |
| 62 | +let instance_ip_cidr = Ipv4Inet::new(instance_ip_addr, 24).unwrap(); |
| 63 | +let ports = vec![Port::new(80, 8080)]; |
| 64 | + |
| 65 | +let request = SetupInstanceRequest::new( |
| 66 | + node_id.to_string(), |
| 67 | + node_ip_addr, |
| 68 | + instance_id.to_string(), |
| 69 | + instance_ip_cidr, |
| 70 | + ports, |
| 71 | +); |
| 72 | +let response = setup_instance(request).unwrap(); |
| 73 | +println!("Instance default interface: {}", response.interface_name); |
| 74 | +println!("Network namespace: {}", response.namespace_name); |
| 75 | +``` |
| 76 | + |
| 77 | +You can also get the namespace's name of a given instance with `get_namespace_name` from `utils` |
| 78 | +module. |
| 79 | + |
| 80 | +```rust |
| 81 | +let instance_id = "instance"; |
| 82 | +let namespace_name = get_namespace_name(instance_id.to_string()); |
| 83 | +println!("Namespace of {}: {}", instance_id, namespace_name); |
| 84 | +``` |
| 85 | + |
| 86 | +### Clean up |
| 87 | + |
| 88 | +To delete CNI and iptables rules of a specific node, use `clean_node` function from `node` module. |
| 89 | + |
| 90 | +```rust |
| 91 | +let node_id = "node"; |
| 92 | +let request = CleanNodeRequest::new(node_id.to_string()); |
| 93 | +clean_node(request).unwrap(); |
55 | 94 | ``` |
56 | 95 |
|
57 | | -### Services |
58 | | - |
59 | | -```protobuf |
60 | | -service Network { |
61 | | - // Create a new virtual inerface and add it to the node CNI |
62 | | - rpc CreateNetworkInterface(CreateNetworkInterfaceRequest) returns (CreateNetworkInterfaceResponse) {} |
63 | | - // Delete a virtual interface |
64 | | - rpc DeleteNetworkInterface(DeleteNetworkInterfaceRequest) returns (Empty) {} |
65 | | - // Create a new virtual network interface (CNI) |
66 | | - rpc Setup(SetupRequest) returns (SetupResponse) {} |
67 | | -} |
| 96 | +Run `clean_instance` function from `instance` module to delete network namespace and interfaces of a |
| 97 | +specific instance. |
| 98 | + |
| 99 | +```rust |
| 100 | +let instance_id = "instance"; |
| 101 | +let instance_ip_addr = Ipv4Addr::from_str("10.0.0.2").unwrap(); |
| 102 | +let instance_ip_cidr = Ipv4Inet::new(instance_ip_addr, 24).unwrap(); |
| 103 | +let ports = vec![Port::new(80, 8080)]; |
| 104 | +let request = CleanInstanceRequest::new( |
| 105 | + instance_id.to_string(), |
| 106 | + ports, |
| 107 | + instance_ip_cidr, |
| 108 | +); |
| 109 | +clean_instance(request).unwrap(); |
68 | 110 | ``` |
0 commit comments