Skip to content

Commit 4e4ee13

Browse files
ananosananos
committed
docs: Add Testbed Setup instructions
Signed-off-by: Anastassios Nanos <ananos@nubificus.co.uk>
1 parent fb20a0d commit 4e4ee13

File tree

3 files changed

+174
-1
lines changed

3 files changed

+174
-1
lines changed

‎docs/.nav.yml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
nav:
22
- Home: index.md
3-
- Quickstart: quickstart.md
3+
- Getting Started:
4+
- Testbed Setup: testbed.md
5+
- Quickstart: quickstart.md
46
- Installation: installation.md
57
- Design:
68
- design/*.md

‎docs/installation.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,15 +7,19 @@ is able to run on a number of distros.
77
We will be installing and setting up each component individually:
88

99
### Core Framework
10+
1011
TBC
1112

1213
### Plugins
14+
1315
TBC
1416

1517
### Mechanisms
18+
1619
TBC
1720

1821
## Install required dependencies
22+
1923
TBC
2024

2125
> Note: Be aware that some instructions might override existing tools and services.

‎docs/testbed.md‎

Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
# Basic Testbed
2+
3+
To install the MLSysOps framework, you will need one or more kubernetes clusters, acting as the compute nodes (cloud / edge layers) and a management node (or cluster), acting as the high-level orchestrator. In the following document we provide the necessary steps to bootstrap an example testbed.
4+
5+
## Quick links
6+
7+
[Prerequisites](#prerequisites)
8+
[Installation Steps](#installation-steps)
9+
* [Setting Up Ansible](#setting-up-ansible)
10+
* [Inventory Configuration](#inventory-configuration)
11+
* [k3s Installation Playbook](#k3s-installation-playbook)
12+
* [Karmada Installation Playbook](#karmada-installation-playbook)
13+
14+
## Prerequisites
15+
Before proceeding, ensure the following:
16+
17+
- **Ansible Installed:** Install Ansible on the control node.
18+
- **SSH Access:** Ensure the control node has SSH access to all target nodes.
19+
- **Python 3 Installed:** Ensure Python 3 is available on all nodes.
20+
- **Supported OS:** The playbooks are tested on Ubuntu 22.04 LTS. Other Linux distributions may require adjustments.
21+
- **Multiple Machines:** At least one machine for the management cluster (Karmada) and others for k3s clusters (master and worker nodes).
22+
23+
Assuming we have bootstrapped a **control node** that will manage the testbed installation, we can proceed.
24+
25+
## Installation Steps
26+
To set up Ansible, run the follow commands on the control node:
27+
28+
1) Update System Packages to latest version.
29+
30+
```
31+
sudo apt update && sudo apt upgrade -y
32+
```
33+
2) Install essential packages that Ansible relies on:
34+
35+
```
36+
sudo apt install -y python3 software-properties-common
37+
```
38+
3) Install Ansible.
39+
40+
```
41+
sudo apt install -y ansible
42+
```
43+
4) After installation, confirm that Ansible is installed and working correctly by checking its version:
44+
```
45+
ansible --version
46+
```
47+
Example output:
48+
49+
```
50+
ansible 2.10.8
51+
config file = None
52+
configured module search path = ['/home/pmavrikos/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
53+
ansible python module location = /usr/lib/python3/dist-packages/ansible
54+
executable location = /usr/bin/ansible
55+
python version = 3.10.12 (main, Feb 4 2025, 14:57:36) [GCC 11.4.0]
56+
```
57+
58+
### Setting Up Ansible
59+
Once Ansible is installed, you need to set up your project by cloning the repository and configuring the necessary files.
60+
61+
62+
**Clone the Repository and navigate to the project directory**
63+
64+
```
65+
git clone https://github.com/mlsysops-eu/mlsysops-framework
66+
cd mlsysops-framework
67+
```
68+
69+
### Inventory Configuration
70+
The inventory.yml file contains the list of target nodes where k3s and Karmada will be installed. Before running the playbooks, update this file with your specific setup.
71+
72+
73+
**Understand the Structure**
74+
75+
The file is divided into:
76+
77+
- **management_cluster:** The machine where Karmada will be installed (usually one node).
78+
- **cluster1:** A k3s cluster with:
79+
- **master_nodes:** Control-plane nodes (you can have one or more for high availability).
80+
- **worker_nodes:** Worker nodes that run workloads.
81+
82+
**Mandatory fields to update:**
83+
84+
- `ansible_host`: Replace `xxxxx` with the IP address of each target node.
85+
- `ansible_user`: Enter the SSH username for logging into the machine
86+
- `ansible_ssh_private_key_file`: Provide the full path to your SSH private key on the control machine.
87+
- `ansible_python_interpreter`: Ensure it points to a valid Python 3 interpreter path on each target node.
88+
- `k3s_cluster_name`: Specify a meaningful cluster name.
89+
- `pod_cidr` and `service_cidr`: Customize network ranges for pods and services (they must not overlap between clusters).
90+
91+
**Example Configuration**
92+
93+
```
94+
all:
95+
children:
96+
management_cluster: # <-- In this vm will be karmada
97+
hosts:
98+
mls00: # <-- Change with your vm name
99+
ansible_host: x.x.x.x # <-- Update with the correct IP address
100+
ansible_user: mlsysops
101+
ansible_ssh_private_key_file: /home/xxxxxx/.ssh/id_rsa # <-- Update
102+
ansible_python_interpreter: /usr/bin/python3
103+
k3s_cluster_name: management
104+
pod_cidr: "x.x.x.x/x"
105+
service_cidr: "x.x.x.x/x"
106+
107+
cluster1:
108+
children:
109+
master_nodes:
110+
hosts:
111+
mls01: # <-- Change with your master node vm name
112+
ansible_host: x.x.x.x # <-- Update with the correct IP address
113+
ansible_user: mlsysops
114+
ansible_ssh_private_key_file: /home/xxxxxxxx/.ssh/id_rsa # <-- Update
115+
ansible_python_interpreter: /usr/bin/python3
116+
k3s_cluster_name: cluster1
117+
pod_cidr: "x.x.x.x/x"
118+
service_cidr: "x.x.x.x/x"
119+
worker_nodes:
120+
hosts:
121+
mls02:
122+
ansible_host: x.x.x.x # <-- Update with the correct IP address
123+
ansible_user: mlsysops
124+
ansible_ssh_private_key_file: /home/xxxxxxxxx/.ssh/id_rsa # <-- Update
125+
ansible_python_interpreter: /usr/bin/python3
126+
k3s_cluster_name: cluster1
127+
mls03:
128+
ansible_host: x.x.x.x # <-- Update with the correct IP address
129+
ansible_user: mlsysops
130+
ansible_ssh_private_key_file: /home/xxxxxxxxxxx/.ssh/id_rsa # <-- Update
131+
ansible_python_interpreter: /usr/bin/python3
132+
k3s_cluster_name: cluster1
133+
```
134+
135+
**Verify**
136+
137+
After editing inventory.yml, save the file and check it for errors. You can test the inventory with:
138+
139+
```
140+
ansible-inventory -i inventory.yml --list
141+
```
142+
This shows all nodes Ansible will target.
143+
144+
### k3s Installation Playbook
145+
The k3s-install.yml playbook automates the deployment of a multi-node k3s cluster.
146+
147+
- Ensure the inventory file is updated before running the playbook.
148+
- Execute the playbook to install k3s across all defined nodes.
149+
- After installation, the kubeconfig file for each cluster is stored at:
150+
```
151+
/home/<ANSIBLE_USER>/.kube/config
152+
```
153+
on the control-plane node.
154+
155+
To run [k3s-install.yml](k3s-install.yml) playbook, use the following command:
156+
```
157+
ansible-playbook -i inventory.yml k3s-install.yml
158+
```
159+
160+
### Karmada Installation Playbook
161+
The karmada-install.yml playbook sets up Karmada, a multi-cluster management system.
162+
163+
To run [karmada-install.yml](karmada-install.yml), playbook, use the following command:
164+
```
165+
ansible-playbook -i inventory.yml karmada-install.yml
166+
```
167+

0 commit comments

Comments
 (0)