Skip to content

Commit 254280f

Browse files
amshindeamshinde
committed
Add scripts to setup CC with qemu on a single node
Scripts create a VM image with TDX support, install required dependencies and start qemu with confidential cluster running on it. Signed-off-by: Archana Shinde <archana.m.shinde@intel.com>
1 parent 2eed62b commit 254280f

4 files changed

Lines changed: 478 additions & 0 deletions

File tree

‎deployment/bare-metal/README.md‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
# Deployment Guide on TD baremetal host
2+
3+
This guide introduces how to setup an Intel TDX host on Ubuntu 24.04 and a TD VM with
4+
a single node kubernetes cluster running on it.
5+
Follow these instructions to setup Intel TDX host, create a TD image, boot the TD and run a
6+
kubernestes cluster within the TD.
7+
8+
### Prerequisite
9+
10+
Instructions are relevant for 4th Generation Intel® Xeon® Scalable Processors with activated Intel® TDX
11+
and all 5th Generation Intel® Xeon® Scalable Processors.
12+
13+
### Setup host
14+
15+
We first need to install ageneric Ubuntu 24.04 server image, install necessay packages to turn
16+
the host OS into an Intel TDX-enabled host OS and enable TDX settings in the BIOS.
17+
Detailed instructions to do so can be found here [setup-tdx-host](https://github.com/canonical/tdx?tab=readme-ov-file#setup-tdx-host).
18+
19+
To setup your host, you will essentially need to do this:
20+
```
21+
$ curl https://raw.githubusercontent.com/canonical/tdx/noble-24.04/setup-tdx-host.sh
22+
$ ./setup-tdx-host.sh
23+
```
24+
25+
Once the above step is completed, you will need to reboot your machine and proceed to change the
26+
BIOS settings to enable TDX.
27+
28+
Go to Socket Configuration > Processor Configuration > TME, TME-MT, TDX.
29+
30+
* Set `Memory Encryption (TME)` to `Enabled`
31+
* Set `Total Memory Encryption Bypass` to `Enabled` (Optional setting for best host OS and regular VM performance.)
32+
* Set `Total Memory Encryption Multi-Tenant (TME-MT)` to `Enabled`
33+
* Set `TME-MT memory integrity` to `Disabled`
34+
* Set `Trust Domain Extension (TDX)` to `Enabled`
35+
* Set `TDX Secure Arbitration Mode Loader (SEAM Loader)` to `Enabled`. (NOTE: This allows loading Intel TDX Loader and Intel TDX Module from the ESP or BIOS.)
36+
* Set `TME-MT/TDX key split` to a non-zero value
37+
38+
Go to `Socket Configuration > Processor Configuration > Software Guard Extension (SGX)`.
39+
40+
* Set `SW Guard Extensions (SGX)` to `Enabled`
41+
42+
Save BIOS settings and boot up. Verify that the host has TDX enabled using dmesg command:
43+
```
44+
$ sudo dmesg | grep -i tdx
45+
[ 1.523617] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.8.0-1004-intel root=UUID=f5524554-48b2-4edf-b0aa-3cebac84b167 ro kvm_intel.tdx=1 nohibernate nomodeset
46+
[ 2.551768] virt/tdx: BIOS enabled: private KeyID range [16, 128)
47+
[ 2.551773] virt/tdx: Disable ACPI S3. Turn off TDX in the BIOS to use ACPI S3.
48+
[ 20.408972] virt/tdx: TDX module: attributes 0x0, vendor_id 0x8086, major_version 2, minor_version 0, build_date 20231112, build_num 635
49+
```
50+
51+
### Setup guest
52+
53+
To setup a guest image with TDX kernel and has all the binaries required for running
54+
a k3s/k8s cluster, run the following script:
55+
56+
```
57+
./setup_cc.sh
58+
```
59+
60+
### Launch a kubernetes cluster
61+
62+
The above step will install a helper script to start a single node kubernetes cluster in the
63+
home directory for the `tdx` user in the guest image.
64+
65+
To ssh into the TD VM:
66+
```
67+
$ curl -LO https://raw.githubusercontent.com/cc-api/cvm-image-rewriter/main/start-virt.sh
68+
$ ./start-virt.sh -i output.qcow2
69+
```
70+
71+
Once you have logged in the TD VM, run the following script to start a single node kubernetes cluster:
72+
```
73+
$ /home/tdx/launch_cc.sh
74+
```

‎deployment/bare-metal/create_k8s_node.sh‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
#!/bin/bash
2+
#
3+
# Copyright (c) 2020 Intel Corporation
4+
#
5+
# SPDX-License-Identifier: Apache-2.0
6+
#
7+
8+
#set -o xtrace
9+
set -o errexit
10+
set -o nounset
11+
set -o pipefail
12+
set -o errtrace
13+
14+
pod_network_cidr=${pod_network_cidr:-"10.244.0.0/16"}
15+
cni_project=${cni_project:-"calico"}
16+
17+
init_cluster() {
18+
if [ -d "$HOME/.kube" ]; then
19+
rm -rf "$HOME/.kube"
20+
fi
21+
22+
sudo bash -c 'modprobe br_netfilter'
23+
sudo bash -c 'modprobe overlay'
24+
sudo bash -c 'swapoff -a'
25+
26+
# initialize cluster
27+
#sudo -E kubeadm init --config=./kubeadm.yaml
28+
kubeadm init --pod-network-cidr=${pod_network_cidr}
29+
30+
mkdir -p "${HOME}/.kube"
31+
cp /etc/kubernetes/admin.conf $HOME/.kube/config
32+
chown $(id -u):$(id -g) $HOME/.kube/config
33+
34+
# taint master node:
35+
kubectl taint nodes --all node-role.kubernetes.io/master-
36+
}
37+
38+
install_cni() {
39+
40+
if [[ $cni_project == "calico" ]]; then
41+
calico_url="https://projectcalico.docs.tigera.io/manifests/calico.yaml"
42+
kubectl apply -f $calico_url
43+
else
44+
flannel_url="https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml"
45+
kubectl apply -f $flannel_url
46+
fi
47+
}
48+
49+
main() {
50+
init_cluster
51+
install_cni
52+
}
53+
54+
main $@

‎deployment/bare-metal/install_dependencies.sh‎

Lines changed: 180 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,180 @@
1+
#!/bin/bash
2+
3+
set -e
4+
5+
http_proxy=${http_proxy:-}
6+
https_proxy=${https_proxy:-}
7+
no_proxy=${no_proxy:-}
8+
9+
function setup_proxy {
10+
cat <<-EOF | sudo tee -a "/tmp/environment"
11+
http_proxy = "${http_proxy}"
12+
https_proxy = "${https_proxy}"
13+
no_proxy = "${no_proxy}"
14+
HTTP_PROXY = "${http_proxy}"
15+
HTTPS_PROXY = "${https_proxy}"
16+
NO_PROXY = "${no_proxy}"
17+
EOF
18+
19+
20+
#cat <<-EOF | sudo tee -a "/etc/profile.d/myenvvar.sh"
21+
cat <<-EOF | sudo tee -a "/tmp/myenvvar.sh"
22+
http_proxy = "${http_proxy}"
23+
https_proxy = "${https_proxy}"
24+
no_proxy = "${no_proxy}"
25+
EOF
26+
27+
sudo sh -c 'systemctl set-environment http_proxy="${http_proxy}"'
28+
sudo sh -c 'systemctl set-environment https_proxy="${https_proxy}"'
29+
sudo sh -c 'systemctl set-environment no_proxy="${no_proxy}"'
30+
}
31+
32+
function install_docker {
33+
# install GPG key
34+
install -m 0755 -d /etc/apt/keyrings
35+
rm -f /etc/apt/keyrings/docker.gpg
36+
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
37+
chmod a+r /etc/apt/keyrings/docker.gpg
38+
39+
# install repo
40+
echo \
41+
"deb [arch=\"$(dpkg --print-architecture)\" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
42+
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
43+
tee /etc/apt/sources.list.d/docker.list > /dev/null
44+
apt-get update > /dev/null
45+
46+
# install docker
47+
apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
48+
systemctl enable docker
49+
50+
add_docker_proxy_for_builds
51+
52+
# Add proxy for docker and containerd. This proxy is used in docker pull
53+
54+
services=("containerd docker")
55+
add_systemd_service_proxy "${services[@]}"
56+
}
57+
58+
function add_docker_proxy_for_builds() {
59+
mkdir -p /home/tdx/.docker
60+
cat <<-EOF | sudo tee "/home/tdx/.docker/config.json"
61+
{
62+
"proxies": {
63+
"default": {
64+
"httpProxy": "${http_proxy}",
65+
"httpsProxy": "${https_proxy}",
66+
"noProxy": "${no_proxy}"
67+
}
68+
}
69+
}
70+
EOF
71+
}
72+
73+
function install_helm {
74+
# install repo
75+
curl -fsSL https://baltocdn.com/helm/signing.asc | gpg --dearmor | tee /usr/share/keyrings/helm.gpg > /dev/null
76+
echo \
77+
"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | \
78+
tee /etc/apt/sources.list.d/helm-stable-debian.list > /dev/null
79+
apt-get update > /dev/null
80+
81+
# install helm
82+
apt-get install -y helm
83+
}
84+
85+
86+
function install_pip {
87+
# install python3-pip
88+
apt install -y python3-pip
89+
}
90+
91+
function install_k3s {
92+
curl -sfL https://get.k3s.io | sh -
93+
94+
#configure proxy
95+
local k3s_env_file="/etc/systemd/system/k3s.service.env"
96+
cat <<-EOF | sudo tee -a $k3s_env_file
97+
HTTP_PROXY="${http_proxy}"
98+
HTTPS_PROXY="${https_proxy}"
99+
NO_PROXY="${no_proxy}"
100+
EOF
101+
102+
}
103+
104+
function install_k8s {
105+
sudo -E bash -c 'apt-get -y clean'
106+
107+
# Install Kubernetes:
108+
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
109+
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
110+
sudo -E apt update
111+
sudo -E apt install -y kubelet kubeadm kubectl
112+
113+
# Packets traversing the bridge should be sent to iptables for processing
114+
echo br_netfilter | sudo -E tee /etc/modules-load.d/k8s.conf
115+
sudo -E bash -c 'echo "net.bridge.bridge-nf-call-ip6tables = 1" > /etc/sysctl.d/k8s.conf'
116+
sudo -E bash -c 'echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.d/k8s.conf'
117+
sudo -E bash -c 'echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.d/k8s.conf'
118+
sudo -E sysctl --system
119+
120+
# disable swap
121+
swapoff -a
122+
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
123+
124+
services=("kubelet")
125+
add_systemd_service_proxy "${services[@]}"
126+
}
127+
128+
function add_systemd_service_proxy() {
129+
local components=("$@")
130+
# Config proxy
131+
local HTTPS_PROXY="$HTTPS_PROXY"
132+
local https_proxy="$https_proxy"
133+
if [ -z "$HTTPS_PROXY" ]; then
134+
HTTPS_PROXY="$https_proxy"
135+
fi
136+
137+
local HTTP_PROXY="$HTTP_PROXY"
138+
local http_proxy="$http_proxy"
139+
if [ -z "$HTTP_PROXY" ]; then
140+
HTTP_PROXY="$http_proxy"
141+
fi
142+
143+
local NO_PROXY="$NO_PROXY"
144+
local no_proxy="$no_proxy"
145+
if [ -z "$NO_PROXY" ]; then
146+
NO_PROXY="$no_proxy"
147+
fi
148+
149+
if [[ -n $HTTP_PROXY ]] || [[ -n $HTTPS_PROXY ]] || [[ -n $NO_PROXY ]]; then
150+
for component in "${components[@]}"; do
151+
echo "component: " "${component}"
152+
mkdir -p /etc/systemd/system/"${component}.service.d"/
153+
tee /etc/systemd/system/"${component}.service.d"/http-proxy.conf <<EOF
154+
[Service]
155+
Environment=\"HTTP_PROXY=${HTTP_PROXY}\"
156+
Environment=\"HTTPS_PROXY=${HTTPS_PROXY}\"
157+
Environment=\"NO_PROXY=${NO_PROXY}\"
158+
EOF
159+
systemctl daemon-reload
160+
systemctl restart ${component}
161+
done
162+
fi
163+
}
164+
165+
166+
167+
function main {
168+
setup_proxy
169+
170+
# install pre-reqs
171+
sudo -E bash -c 'apt-get update && sudo -E apt install -y curl'
172+
173+
install_docker
174+
install_helm
175+
#install_pip
176+
install_k8s
177+
install_k3s
178+
}
179+
180+
main $@

0 commit comments

Comments
 (0)