Skip to content

Commit 8c07a99

Browse files
amshindeamshinde
committed
Add scripts to setup CC with qemu on a single node
Scripts create a VM image with TDX support, install required dependencies and start qemu with confidential cluster running on it. Signed-off-by: Archana Shinde <archana.m.shinde@intel.com>
1 parent 2eed62b commit 8c07a99

4 files changed

Lines changed: 484 additions & 0 deletions

File tree

‎deployment/bare-metal/README.md‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
# Deployment Guide on TD baremetal host
2+
3+
This guide introduces how to setup an Intel TDX host on Ubuntu 24.04 and a TD VM with
4+
a single node kubernetes cluster running on it.
5+
Follow these instructions to setup Intel TDX host, create a TD image, boot the TD and run a
6+
kubernetes cluster within the TD.
7+
8+
### Prerequisite
9+
10+
Instructions are relevant for 4th Generation Intel® Xeon® Scalable Processors with activated Intel® TDX
11+
and all 5th Generation Intel® Xeon® Scalable Processors.
12+
13+
### Setup host
14+
15+
We first need to install a generic Ubuntu 24.04 server image, install necessary packages to turn
16+
the host OS into an Intel TDX-enabled host OS and enable TDX settings in the BIOS.
17+
Detailed instructions to do so can be found here [setup-tdx-host](https://github.com/canonical/tdx?tab=readme-ov-file#setup-tdx-host).
18+
19+
To setup your host, you will essentially need to do this:
20+
```
21+
$ curl https://raw.githubusercontent.com/canonical/tdx/noble-24.04/setup-tdx-host.sh
22+
$ ./setup-tdx-host.sh
23+
```
24+
25+
Once the above step is completed, you will need to reboot your machine and proceed to change the
26+
BIOS settings to enable TDX.
27+
28+
Go to Socket Configuration > Processor Configuration > TME, TME-MT, TDX.
29+
30+
* Set `Memory Encryption (TME)` to `Enabled`
31+
* Set `Total Memory Encryption Bypass` to `Enabled` (Optional setting for best host OS and regular VM performance.)
32+
* Set `Total Memory Encryption Multi-Tenant (TME-MT)` to `Enabled`
33+
* Set `TME-MT memory integrity` to `Disabled`
34+
* Set `Trust Domain Extension (TDX)` to `Enabled`
35+
* Set `TDX Secure Arbitration Mode Loader (SEAM Loader)` to `Enabled`. (NOTE: This allows loading Intel TDX Loader and Intel TDX Module from the ESP or BIOS.)
36+
* Set `TME-MT/TDX key split` to a non-zero value
37+
38+
Go to `Socket Configuration > Processor Configuration > Software Guard Extension (SGX)`.
39+
40+
* Set `SW Guard Extensions (SGX)` to `Enabled`
41+
42+
Save BIOS settings and boot up. Verify that the host has TDX enabled using dmesg command:
43+
```
44+
$ sudo dmesg | grep -i tdx
45+
[ 1.523617] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.8.0-1004-intel root=UUID=f5524554-48b2-4edf-b0aa-3cebac84b167 ro kvm_intel.tdx=1 nohibernate nomodeset
46+
[ 2.551768] virt/tdx: BIOS enabled: private KeyID range [16, 128)
47+
[ 2.551773] virt/tdx: Disable ACPI S3. Turn off TDX in the BIOS to use ACPI S3.
48+
[ 20.408972] virt/tdx: TDX module: attributes 0x0, vendor_id 0x8086, major_version 2, minor_version 0, build_date 20231112, build_num 635
49+
```
50+
51+
### Setup guest
52+
53+
To setup a guest image with TDX kernel and has all the binaries required for running
54+
a k3s/k8s cluster, run the following script:
55+
56+
```
57+
./setup_cc.sh
58+
```
59+
60+
After running the script, you should see an image with the name `tdx-guest-ubuntu-24.04-intel.qcow2`
61+
generated in the current directory. The image is setup with user `tdx` with password `123456`.
62+
63+
### Launch a kubernetes cluster
64+
65+
The above step will install a helper script to start a single node kubernetes cluster in the
66+
home directory for the `tdx` user in the guest image.
67+
68+
To ssh into the TD VM:
69+
```
70+
$ curl -LO https://raw.githubusercontent.com/cc-api/cvm-image-rewriter/main/start-virt.sh
71+
$ ./start-virt.sh -i tdx-guest-ubuntu-24.04-intel.qcow2
72+
```
73+
74+
Once you have logged in the TD VM, run the following script to start a single node kubernetes cluster:
75+
```
76+
$ /home/tdx/create_k8s_node.sh
77+
```

‎deployment/bare-metal/create_k8s_node.sh‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
#!/bin/bash
2+
#
3+
# Copyright (c) 2024 Intel Corporation
4+
#
5+
# SPDX-License-Identifier: Apache-2.0
6+
#
7+
8+
#set -o xtrace
9+
set -o errexit
10+
set -o nounset
11+
set -o pipefail
12+
set -o errtrace
13+
14+
pod_network_cidr=${pod_network_cidr:-"10.244.0.0/16"}
15+
cni_project=${cni_project:-"calico"}
16+
17+
init_cluster() {
18+
if [ -d "$HOME/.kube" ]; then
19+
rm -rf "$HOME/.kube"
20+
fi
21+
22+
sudo bash -c 'modprobe br_netfilter'
23+
sudo bash -c 'modprobe overlay'
24+
sudo bash -c 'swapoff -a'
25+
26+
# initialize cluster
27+
#sudo -E kubeadm init --config=./kubeadm.yaml
28+
kubeadm init --pod-network-cidr=${pod_network_cidr}
29+
30+
mkdir -p "${HOME}/.kube"
31+
cp /etc/kubernetes/admin.conf $HOME/.kube/config
32+
chown $(id -u):$(id -g) $HOME/.kube/config
33+
34+
# taint master node:
35+
kubectl taint nodes --all node-role.kubernetes.io/master-
36+
}
37+
38+
install_cni() {
39+
40+
if [[ $cni_project == "calico" ]]; then
41+
calico_url="https://projectcalico.docs.tigera.io/manifests/calico.yaml"
42+
kubectl apply -f $calico_url
43+
else
44+
flannel_url="https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml"
45+
kubectl apply -f $flannel_url
46+
fi
47+
}
48+
49+
main() {
50+
init_cluster
51+
install_cni
52+
}
53+
54+
main $@

‎deployment/bare-metal/install_dependencies.sh‎

Lines changed: 190 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,190 @@
1+
#!/bin/bash
2+
#
3+
# Copyright (c) 2024 Intel Corporation
4+
#
5+
# SPDX-License-Identifier: Apache-2.0
6+
#
7+
8+
set -e
9+
10+
http_proxy=${http_proxy:-}
11+
https_proxy=${https_proxy:-}
12+
no_proxy=${no_proxy:-}
13+
14+
function setup_proxy {
15+
cat <<-EOF | sudo tee -a "/etc/environment"
16+
http_proxy="${http_proxy}"
17+
https_proxy="${https_proxy}"
18+
no_proxy="${no_proxy}"
19+
HTTP_PROXY="${http_proxy}"
20+
HTTPS_PROXY="${https_proxy}"
21+
NO_PROXY="${no_proxy}"
22+
EOF
23+
24+
25+
cat <<-EOF | sudo tee -a "/etc/profile.d/myenvvar.sh"
26+
export http_proxy="${http_proxy}"
27+
export https_proxy="${https_proxy}"
28+
export no_proxy="${no_proxy}"
29+
EOF
30+
31+
sudo sh -c 'systemctl set-environment http_proxy="${http_proxy}"'
32+
sudo sh -c 'systemctl set-environment https_proxy="${https_proxy}"'
33+
sudo sh -c 'systemctl set-environment no_proxy="${no_proxy}"'
34+
}
35+
36+
function install_docker {
37+
# install GPG key
38+
install -m 0755 -d /etc/apt/keyrings
39+
rm -f /etc/apt/keyrings/docker.gpg
40+
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
41+
chmod a+r /etc/apt/keyrings/docker.gpg
42+
43+
# install repo
44+
echo \
45+
"deb [arch=\"$(dpkg --print-architecture)\" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
46+
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
47+
tee /etc/apt/sources.list.d/docker.list > /dev/null
48+
apt-get update > /dev/null
49+
50+
# install docker
51+
apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
52+
systemctl enable docker
53+
54+
add_docker_proxy_for_builds
55+
56+
# Add proxy for docker and containerd. This proxy is used in docker pull
57+
58+
services=("containerd" "docker")
59+
add_systemd_service_proxy "${services[@]}"
60+
}
61+
62+
function add_docker_proxy_for_builds() {
63+
mkdir -p /home/tdx/.docker
64+
cat <<-EOF | sudo tee "/home/tdx/.docker/config.json"
65+
{
66+
"proxies": {
67+
"default": {
68+
"httpProxy": "${http_proxy}",
69+
"httpsProxy": "${https_proxy}",
70+
"noProxy": "${no_proxy}"
71+
}
72+
}
73+
}
74+
EOF
75+
}
76+
77+
function install_helm {
78+
# install repo
79+
curl -fsSL https://baltocdn.com/helm/signing.asc | gpg --dearmor | tee /usr/share/keyrings/helm.gpg > /dev/null
80+
echo \
81+
"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | \
82+
tee /etc/apt/sources.list.d/helm-stable-debian.list > /dev/null
83+
apt-get update > /dev/null
84+
85+
# install helm
86+
apt-get install -y helm
87+
}
88+
89+
90+
function install_pip {
91+
# install python3-pip
92+
apt install -y python3-pip
93+
}
94+
95+
function install_k3s {
96+
curl -sfL https://get.k3s.io | sh -
97+
98+
#configure proxy
99+
local k3s_env_file="/etc/systemd/system/k3s.service.env"
100+
cat <<-EOF | sudo tee -a $k3s_env_file
101+
HTTP_PROXY="${http_proxy}"
102+
HTTPS_PROXY="${https_proxy}"
103+
NO_PROXY="${no_proxy}"
104+
EOF
105+
106+
}
107+
108+
function install_k8s {
109+
sudo -E bash -c 'apt-get -y clean'
110+
sudo -E bash -c 'apt-get purge kubeadm kubectl kubelet' | true
111+
112+
sudo apt-get install -y wget apt-transport-https ca-certificates curl gnupg
113+
114+
wget https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key
115+
gpg --no-default-keyring --keyring /tmp/k8s_keyring.gpg --import Release.key
116+
gpg --no-default-keyring --keyring /tmp/k8s_keyring.gpg --export > /tmp/k8s.gpg
117+
sudo bash -c 'mv /tmp/k8s.gpg /etc/apt/trusted.gpg.d/'
118+
119+
echo 'deb https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
120+
sudo -E apt-get update
121+
sudo -E apt install -y kubelet kubeadm kubectl
122+
123+
# Packets traversing the bridge should be sent to iptables for processing
124+
echo br_netfilter | sudo -E tee /etc/modules-load.d/k8s.conf
125+
sudo -E bash -c 'echo "net.bridge.bridge-nf-call-ip6tables = 1" > /etc/sysctl.d/k8s.conf'
126+
sudo -E bash -c 'echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.d/k8s.conf'
127+
sudo -E bash -c 'echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.d/k8s.conf'
128+
sudo -E sysctl --system
129+
130+
# disable swap
131+
swapoff -a
132+
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
133+
134+
services=("kubelet")
135+
add_systemd_service_proxy "${services[@]}"
136+
}
137+
138+
function add_systemd_service_proxy() {
139+
local components=("$@")
140+
# Config proxy
141+
local HTTPS_PROXY="$HTTPS_PROXY"
142+
local https_proxy="$https_proxy"
143+
if [ -z "$HTTPS_PROXY" ]; then
144+
HTTPS_PROXY="$https_proxy"
145+
fi
146+
147+
local HTTP_PROXY="$HTTP_PROXY"
148+
local http_proxy="$http_proxy"
149+
if [ -z "$HTTP_PROXY" ]; then
150+
HTTP_PROXY="$http_proxy"
151+
fi
152+
153+
local NO_PROXY="$NO_PROXY"
154+
local no_proxy="$no_proxy"
155+
if [ -z "$NO_PROXY" ]; then
156+
NO_PROXY="$no_proxy"
157+
fi
158+
159+
if [[ -n $HTTP_PROXY ]] || [[ -n $HTTPS_PROXY ]] || [[ -n $NO_PROXY ]]; then
160+
for component in "${components[@]}"; do
161+
echo "component: " "${component}"
162+
mkdir -p /etc/systemd/system/"${component}.service.d"/
163+
tee /etc/systemd/system/"${component}.service.d"/http-proxy.conf <<EOF
164+
[Service]
165+
Environment=\"HTTP_PROXY=${HTTP_PROXY}\"
166+
Environment=\"HTTPS_PROXY=${HTTPS_PROXY}\"
167+
Environment=\"NO_PROXY=${NO_PROXY}\"
168+
EOF
169+
systemctl daemon-reload
170+
systemctl restart ${component}
171+
done
172+
fi
173+
}
174+
175+
176+
177+
function main {
178+
setup_proxy
179+
180+
# install pre-reqs
181+
sudo -E bash -c 'apt-get update && sudo -E apt install -y curl'
182+
183+
install_docker
184+
install_helm
185+
install_pip
186+
install_k8s
187+
install_k3s
188+
}
189+
190+
main $@

0 commit comments

Comments
 (0)