Skip to content

Commit 81b18ef

Browse files
panosmaurikosananos
panosmaurikos
authored and
ananos
committed
feat: Add ansible files to bootstrap testbed
- Add karmada ansible logic - Add retry mechanism - Fix inventory.yml and karmada-install.yml Signed-off-by: Panos Mavrikos <pmavrikos@nubificus.co.uk>
1 parent 419f32f commit 81b18ef

7 files changed

Lines changed: 574 additions & 1 deletion

File tree

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
values.yaml

‎README.md‎

Lines changed: 0 additions & 1 deletion
This file was deleted.

‎orchestrators/README.md‎

Lines changed: 172 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,172 @@
1+
Ansible Playbook for k3s and Karmada Installation
2+
=================================================
3+
4+
This document provides a step-by-step guide to using the provided Ansible playbook to install and configure k3s across multiple clusters. Additionally, it includes instructions for setting up Karmada, a multi-cluster management tool.
5+
6+
7+
8+
## Table of Contents
9+
10+
1. [Prerequisites](#prerequisites)
11+
2. [Installation Steps](#installation-steps)
12+
* [Setting Up Ansible](#setting-up-ansible)
13+
* [Inventory Configuration](#inventory-configuration)
14+
* [k3s Installation Playbook](#k3s-installation-playbook)
15+
* [Karmada Installation Playbook](#karmada-installation-playbook)
16+
17+
18+
19+
## Prerequisites
20+
Before proceeding, ensure the following:
21+
22+
- **Ansible Installed:** Install Ansible on the control node.
23+
- **SSH Access:** Ensure the control node has SSH access to all target nodes.
24+
- **Python 3 Installed:** Ensure Python 3 is available on all nodes.
25+
- **Supported OS:** The playbooks are tested on Ubuntu 22.04 LTS. Other Linux distributions may require adjustments.
26+
- **Multiple Machines:** At least one machine for the management cluster (Karmada) and others for k3s clusters (master and worker nodes).
27+
28+
## Installation Steps
29+
To set up Ansible, run the follow commands:
30+
31+
1) Update System Packages to latest version.
32+
33+
```
34+
sudo apt update && sudo apt upgrade -y
35+
```
36+
2) Install essential packages that Ansible relies on:
37+
38+
```
39+
sudo apt install -y python3 software-properties-common
40+
```
41+
3) Install Ansible.
42+
43+
```
44+
sudo apt install -y ansible
45+
```
46+
4) After installation, confirm that Ansible is installed and working correctly by checking its version:
47+
```
48+
ansible --version
49+
```
50+
Example output:
51+
52+
```
53+
ansible 2.10.8
54+
config file = None
55+
configured module search path = ['/home/pmavrikos/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
56+
ansible python module location = /usr/lib/python3/dist-packages/ansible
57+
executable location = /usr/bin/ansible
58+
python version = 3.10.12 (main, Feb 4 2025, 14:57:36) [GCC 11.4.0]
59+
```
60+
61+
### Setting Up Ansible
62+
Once Ansible is installed, you need to set up your project by cloning the repository and configuring the necessary files.
63+
64+
65+
**Clone the Repository and navigate to the project directory**
66+
67+
```
68+
git clone https://github.com/nubificus/mlsysopsansible.git
69+
cd mlsysopsansible
70+
```
71+
72+
### Inventory Configuration
73+
The inventory.yml file contains the list of target nodes where k3s and Karmada will be installed. Before running the playbooks, update this file with your specific setup.
74+
75+
76+
**Understand the Structure**
77+
78+
The file is divided into:
79+
80+
- **management_cluster:** The machine where Karmada will be installed (usually one node).
81+
- **cluster1:** A k3s cluster with:
82+
- **master_nodes:** Control-plane nodes (you can have one or more for high availability).
83+
- **worker_nodes:** Worker nodes that run workloads.
84+
85+
**Mandatory fields to update:**
86+
87+
- `ansible_host`: Replace `xxxxx` with the IP address of each target node.
88+
- `ansible_user`: Enter the SSH username for logging into the machine
89+
- `ansible_ssh_private_key_file`: Provide the full path to your SSH private key on the control machine.
90+
- `ansible_python_interpreter`: Ensure it points to a valid Python 3 interpreter path on each target node.
91+
- `k3s_cluster_name`: Specify a meaningful cluster name.
92+
- `pod_cidr` and `service_cidr`: Customize network ranges for pods and services (they must not overlap between clusters).
93+
94+
**Example Configuration**
95+
96+
```
97+
all:
98+
children:
99+
management_cluster: # <-- In this vm will be karmada
100+
hosts:
101+
mls00: # <-- Change with your vm name
102+
ansible_host: x.x.x.x # <-- Update with the correct IP address
103+
ansible_user: mlsysops
104+
ansible_ssh_private_key_file: /home/xxxxxx/.ssh/id_rsa # <-- Update
105+
ansible_python_interpreter: /usr/bin/python3
106+
k3s_cluster_name: management
107+
pod_cidr: "x.x.x.x/x"
108+
service_cidr: "x.x.x.x/x"
109+
110+
cluster1:
111+
children:
112+
master_nodes:
113+
hosts:
114+
mls01: # <-- Change with your master node vm name
115+
ansible_host: x.x.x.x # <-- Update with the correct IP address
116+
ansible_user: mlsysops
117+
ansible_ssh_private_key_file: /home/xxxxxxxx/.ssh/id_rsa # <-- Update
118+
ansible_python_interpreter: /usr/bin/python3
119+
k3s_cluster_name: cluster1
120+
pod_cidr: "x.x.x.x/x"
121+
service_cidr: "x.x.x.x/x"
122+
worker_nodes:
123+
hosts:
124+
mls02:
125+
ansible_host: x.x.x.x # <-- Update with the correct IP address
126+
ansible_user: mlsysops
127+
ansible_ssh_private_key_file: /home/xxxxxxxxx/.ssh/id_rsa # <-- Update
128+
ansible_python_interpreter: /usr/bin/python3
129+
k3s_cluster_name: cluster1
130+
mls03:
131+
ansible_host: x.x.x.x # <-- Update with the correct IP address
132+
ansible_user: mlsysops
133+
ansible_ssh_private_key_file: /home/xxxxxxxxxxx/.ssh/id_rsa # <-- Update
134+
ansible_python_interpreter: /usr/bin/python3
135+
k3s_cluster_name: cluster1
136+
```
137+
138+
**Verify**
139+
140+
After editing inventory.yml, save the file and check it for errors. You can test the inventory with:
141+
142+
```
143+
ansible-inventory -i inventory.yml --list
144+
```
145+
This shows all nodes Ansible will target.
146+
147+
### k3s Installation Playbook
148+
The k3s-install.yml playbook automates the deployment of a multi-node k3s cluster.
149+
150+
- Ensure the inventory file is updated before running the playbook.
151+
- Execute the playbook to install k3s across all defined nodes.
152+
- After installation, the kubeconfig file for each cluster is stored at:
153+
```
154+
/home/<ANSIBLE_USER>/.kube/config
155+
```
156+
on the control-plane node.
157+
158+
To run [k3s-install.yml](k3s-install.yml) playbook, use the following command:
159+
```
160+
ansible-playbook -i inventory.yml k3s-install.yml
161+
```
162+
163+
### Karmada Installation Playbook
164+
The karmada-install.yml playbook sets up Karmada, a multi-cluster management system.
165+
166+
To run [karmada-install.yml](karmada-install.yml), playbook, use the following command:
167+
```
168+
ansible-playbook -i inventory.yml karmada-install.yml
169+
```
170+
171+
172+

‎orchestrators/ansible.cfg‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
[defaults]
2+
host_key_checking = False

‎orchestrators/inventory.yml‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
all:
2+
children:
3+
management_cluster:
4+
hosts:
5+
mls00:
6+
ansible_host: 192.168.5.79
7+
ansible_user: mlsysops
8+
ansible_ssh_private_key_file: /home/mlsysops/.ssh/id_rsa
9+
ansible_python_interpreter: /usr/bin/python3
10+
k3s_cluster_name: management
11+
pod_cidr: "10.10.0.0/16"
12+
service_cidr: "10.11.0.0/16"
13+
14+
cluster1:
15+
children:
16+
master_nodes:
17+
hosts:
18+
mls01:
19+
ansible_host: 192.168.5.25
20+
ansible_user: mlsysops
21+
ansible_ssh_private_key_file: /home/mlsysops/.ssh/id_rsa
22+
ansible_python_interpreter: /usr/bin/python3
23+
k3s_cluster_name: cluster1
24+
pod_cidr: "10.12.0.0/16"
25+
service_cidr: "10.13.0.0/16"
26+
worker_nodes:
27+
hosts:
28+
mls02:
29+
ansible_host: 192.168.5.55
30+
ansible_user: mlsysops
31+
ansible_ssh_private_key_file: /home/mlsysops/.ssh/id_rsa
32+
ansible_python_interpreter: /usr/bin/python3
33+
k3s_cluster_name: cluster1
34+
mls03:
35+
ansible_host: 192.168.5.80
36+
ansible_user: mlsysops
37+
ansible_ssh_private_key_file: /home/mlsysops/.ssh/id_rsa
38+
ansible_python_interpreter: /usr/bin/python3
39+
k3s_cluster_name: cluster1
40+
# cluster2:
41+
# children:
42+
# master_nodes:
43+
# hosts:
44+
# mls04:
45+
# ansible_host: 192.168.5.56
46+
# ansible_user: mlsysops
47+
# ansible_ssh_private_key_file: /home/mlsysops/.ssh/id_rsa
48+
# ansible_python_interpreter: /usr/bin/python3
49+
# k3s_cluster_name: cluster2
50+
# pod_cidr: "10.14.0.0/16"
51+
# service_cidr: "10.15.0.0/16"
52+
#worker_nodes:
53+
# hosts: {}

‎orchestrators/k3s-install.yml‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
---
2+
- name: Install k3s on all clusters
3+
hosts: all
4+
become: yes
5+
become_user: root
6+
become_method: sudo
7+
gather_facts: true
8+
9+
vars:
10+
k3s_version: "v1.31.6+k3s1"
11+
12+
tasks:
13+
- name: Install required system packages
14+
apt:
15+
pkg:
16+
- apt-transport-https
17+
- ca-certificates
18+
- curl
19+
- software-properties-common
20+
state: latest
21+
update_cache: true
22+
23+
- name: Install k3s on master nodes
24+
shell: >
25+
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION={{ k3s_version }} sh -s - server
26+
--cluster-cidr={{ hostvars[inventory_hostname].pod_cidr }}
27+
--service-cidr={{ hostvars[inventory_hostname].service_cidr }}
28+
--node-ip {{ ansible_host }}
29+
args:
30+
when: inventory_hostname in groups['master_nodes'] or 'management_cluster' in group_names
31+
32+
- name: Get k3s node token from masters
33+
ansible.builtin.slurp:
34+
src: /var/lib/rancher/k3s/server/node-token
35+
register: node_token
36+
when: inventory_hostname in groups['master_nodes'] or 'management_cluster' in group_names
37+
38+
- name: Install k3s on worker nodes
39+
shell: >
40+
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION={{ k3s_version }} sh -s - agent
41+
--server https://{{ hostvars[groups['master_nodes'][0]]['ansible_host'] }}:6443
42+
--token {{ hostvars[groups['master_nodes'][0]]['node_token']['content'] | b64decode | trim }}
43+
--node-ip {{ ansible_host }}
44+
args:
45+
when: inventory_hostname in groups['worker_nodes']
46+
47+
- name: Ensure k3s config is readable
48+
file:
49+
path: /etc/rancher/k3s/k3s.yaml
50+
mode: '0644'
51+
when: inventory_hostname in groups['master_nodes'] or 'management_cluster' in group_names
52+
53+
- name: Ensure .kube directory exists for user
54+
file:
55+
path: "/home/{{ ansible_user }}/.kube"
56+
state: directory
57+
owner: "{{ ansible_user }}"
58+
group: "{{ ansible_user }}"
59+
mode: '0700'
60+
when: inventory_hostname in groups['master_nodes'] or 'management_cluster' in group_names
61+
62+
- name: Copy k3s config to user's kube config
63+
copy:
64+
src: /etc/rancher/k3s/k3s.yaml
65+
dest: "/home/{{ ansible_user }}/.kube/config"
66+
owner: "{{ ansible_user }}"
67+
group: "{{ ansible_user }}"
68+
mode: '0644'
69+
remote_src: true
70+
when: inventory_hostname in groups['master_nodes'] or 'management_cluster' in group_names

0 commit comments

Comments
 (0)