Skip to content

Commit a95fae5

Browse files
clstokesrajsinghtech
clstokes
and
rajsinghtech
authored
terraform: add aws-eks-operator (#44)
Co-authored-by: Raj Singh <rajsinghcpre@gmail.com>
1 parent 7021989 commit a95fae5

File tree

9 files changed

+415
-6
lines changed

9 files changed

+415
-6
lines changed

‎.github/workflows/terraform-examples.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ on:
99

1010
jobs:
1111

12-
terraform-tflint:
12+
terraform-check-tflint:
1313
runs-on: ubuntu-latest
1414
steps:
1515
- name: Check out code

‎Makefile‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,18 @@
11
default: help
22

3-
.PHONY: terraform-tflint
4-
terraform-tflint: ## Run 'terraform-tflint' github actions with https://github.com/nektos/act
5-
act -j terraform-tflint
3+
.PHONY: terraform-check-tflint
4+
terraform-check-tflint: ## Run 'terraform-check-tflint' workflow with https://github.com/nektos/act
5+
act -j terraform-check-tflint
66

7-
.PHONY: check-terraform-examples
8-
terraform-check-examples: ## Run specific 'check' github actions with https://github.com/nektos/act
7+
.PHONY: terraform-check-examples
8+
terraform-check-examples: ## Run specific 'check' workflow with https://github.com/nektos/act
99
act -j terraform-check-fmt
1010
act -j terraform-check-variables-tailscale-install-scripts
1111

12+
.PHONY: terraform-fmt
13+
terraform-fmt: ## Run 'terraform-fmt' workflow with https://github.com/nektos/act
14+
@terraform fmt -recursive
15+
1216
.PHONY: help
1317
help: ## Display this information. Default target.
1418
@echo "Valid targets:"

‎terraform/aws/aws-eks-operator/README.md‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
# aws-eks-operator
2+
3+
This example creates the following:
4+
5+
- a VPC and related resources including a NAT Gateway
6+
- an EKS cluster with a managed node group
7+
- a Kubernetes namespace for the [Tailscale operator](https://tailscale.com/kb/1236/kubernetes-operator)
8+
- the Tailscale Kubernetes Operator deployed via [Helm](https://tailscale.com/kb/1236/kubernetes-operator#helm)
9+
- a [high availability API server proxy](https://tailscale.com/kb/1437/kubernetes-operator-api-server-proxy#configuring-a-high-availability-api-server-proxy)
10+
11+
## Considerations
12+
13+
- The EKS cluster is configured with both public and private API server access for flexibility
14+
- The Tailscale operator is deployed in a dedicated `tailscale` namespace
15+
- The operator will create a Tailscale device for API server proxy access
16+
- Any additional Tailscale resources (like ingress controllers) created by the operator will appear in your Tailnet
17+
18+
## Prerequisites
19+
20+
- Follow the [Kubernetes Operator prerequisites](https://tailscale.com/kb/1236/kubernetes-operator#prerequisites).
21+
- For the [high availability API server proxy](https://tailscale.com/kb/1437/kubernetes-operator-api-server-proxy#configuring-a-high-availability-api-server-proxy):
22+
- The configuration as-is currently only works on macOS or Linux clients. Remove or comment out the `null_resource` provisioners that deploy `tailscale-api-server-ha-proxy.yaml` to run from other platforms.
23+
- Requires the [kubectl CLI](https://kubernetes.io/docs/reference/kubectl/) and [AWS CLI](https://aws.amazon.com/cli/).
24+
25+
26+
## To use
27+
28+
Follow the documentation to configure the Terraform providers:
29+
30+
- [AWS](https://registry.terraform.io/providers/hashicorp/aws/latest/docs)
31+
32+
### Configure variables
33+
34+
Create a `terraform.tfvars` file with your Tailscale OAuth credentials:
35+
36+
```hcl
37+
tailscale_oauth_client_id = "your-oauth-client-id"
38+
tailscale_oauth_client_secret = "your-oauth-client-secret"
39+
```
40+
41+
### Deploy
42+
43+
```shell
44+
terraform init
45+
terraform apply
46+
```
47+
48+
#### Verify deployment
49+
50+
After deployment, configure kubectl to access your cluster:
51+
52+
```shell
53+
aws eks update-kubeconfig --region $AWS_REGION --name $(terraform output -raw cluster_name)
54+
```
55+
56+
Check that the Tailscale operator is running:
57+
58+
```shell
59+
kubectl get pods -n tailscale
60+
kubectl logs -n tailscale -l app=operator
61+
```
62+
63+
#### Verify connectivity via the [API server proxy](https://tailscale.com/kb/1437/kubernetes-operator-api-server-proxy)
64+
65+
After deployment, configure kubectl to access your cluster using Tailscale:
66+
67+
```shell
68+
tailscale configure kubeconfig $(terraform output -raw ha_proxy_service_name)
69+
```
70+
71+
```shell
72+
kubectl get pods -n tailscale
73+
```
74+
75+
## To destroy
76+
77+
```shell
78+
terraform destroy
79+
80+
# remove leftover Tailscale devices at https://login.tailscale.com/admin/machines and services at https://login.tailscale.com/admin/services
81+
```
82+
83+
## Limitations
84+
85+
- The [HA API server proxy](https://tailscale.com/kb/1437/kubernetes-operator-api-server-proxy#configuring-a-high-availability-api-server-proxy) is deployed using a [terraform null_resource](https://registry.terraform.io/providers/hashicorp/null/latest/docs/resources/resource) instead of [kubernetes_manifest](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/manifest.html) due to a Terraform limitation that results in `cannot create REST client: no client config` errors on first run.

‎terraform/aws/aws-eks-operator/data.tf‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
data "aws_region" "current" {}
2+
3+
data "aws_eks_cluster_versions" "latest" {
4+
default_only = true
5+
}
6+
7+
data "aws_eks_cluster_auth" "this" {
8+
name = module.eks.cluster_name
9+
}

‎terraform/aws/aws-eks-operator/main.tf‎

Lines changed: 196 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,196 @@
1+
locals {
2+
name = "example-${basename(path.cwd)}"
3+
name_with_random_suffix = "${local.name}-${random_integer.operator_name_suffix.result}"
4+
5+
aws_tags = {
6+
Name = local.name
7+
}
8+
9+
# Modify these to use your own VPC
10+
vpc_id = module.vpc.vpc_id
11+
subnet_ids = module.vpc.private_subnets
12+
13+
# EKS cluster configuration
14+
cluster_name = local.name_with_random_suffix
15+
cluster_version = data.aws_eks_cluster_versions.latest.cluster_versions[0].cluster_version
16+
node_instance_type = "t3.medium"
17+
desired_size = 2
18+
max_size = 2
19+
min_size = 1
20+
21+
# Tailscale Operator configuration
22+
namespace_name = "tailscale"
23+
operator_name = local.name_with_random_suffix
24+
operator_version = "1.92.4"
25+
tailscale_oauth_client_id = var.tailscale_oauth_client_id
26+
tailscale_oauth_client_secret = var.tailscale_oauth_client_secret
27+
28+
enable_ha_proxy_service = true
29+
ha_proxy_service_name = "${helm_release.tailscale_operator.name}-ha"
30+
}
31+
32+
# This isn't required but helps avoid conflicts and Let's Encrypt throttling to make testing and iterating easier.
33+
resource "random_integer" "operator_name_suffix" {
34+
min = 100
35+
max = 999
36+
}
37+
38+
# Remove this to use your own VPC.
39+
module "vpc" {
40+
source = "../internal-modules/aws-vpc"
41+
42+
name = local.name
43+
tags = local.aws_tags
44+
}
45+
46+
module "eks" {
47+
source = "terraform-aws-modules/eks/aws"
48+
version = ">= 21.0, < 22.0"
49+
50+
name = local.cluster_name
51+
kubernetes_version = local.cluster_version
52+
53+
tags = local.aws_tags
54+
55+
addons = {
56+
coredns = {}
57+
eks-pod-identity-agent = {
58+
before_compute = true
59+
}
60+
kube-proxy = {}
61+
vpc-cni = {
62+
before_compute = true
63+
}
64+
}
65+
66+
# Once the Tailscale operator is installed, `endpoint_public_access` can be disabled.
67+
# This is left enabled for the sake of easy adoption.
68+
endpoint_public_access = true
69+
70+
# Optional: Adds the current caller identity as an administrator via cluster access entry
71+
enable_cluster_creator_admin_permissions = true
72+
73+
vpc_id = local.vpc_id
74+
subnet_ids = local.subnet_ids
75+
76+
eks_managed_node_groups = {
77+
main = {
78+
# Truncate the node group name to 20 characters to comply with AWS/EKS
79+
# node group naming length constraints.
80+
name = substr(local.name, 0, 20)
81+
instance_types = [local.node_instance_type]
82+
83+
labels = {}
84+
85+
launch_template_name = local.name
86+
launch_template_tags = local.aws_tags
87+
88+
desired_size = local.desired_size
89+
max_size = local.max_size
90+
min_size = local.min_size
91+
}
92+
}
93+
}
94+
95+
resource "kubernetes_namespace_v1" "tailscale_operator" {
96+
provider = kubernetes.this
97+
98+
metadata {
99+
name = local.namespace_name
100+
labels = {
101+
"pod-security.kubernetes.io/enforce" = "privileged"
102+
}
103+
}
104+
105+
depends_on = [
106+
module.eks,
107+
]
108+
}
109+
110+
#
111+
# https://tailscale.com/kb/1236/kubernetes-operator#helm
112+
#
113+
resource "helm_release" "tailscale_operator" {
114+
provider = helm.this
115+
116+
name = local.operator_name
117+
namespace = kubernetes_namespace_v1.tailscale_operator.metadata[0].name
118+
119+
repository = "https://pkgs.tailscale.com/helmcharts"
120+
chart = "tailscale-operator"
121+
version = local.operator_version
122+
123+
values = [
124+
yamlencode({
125+
operatorConfig = {
126+
image = {
127+
tag = "v${local.operator_version}"
128+
}
129+
hostname = local.operator_name
130+
}
131+
apiServerProxyConfig = {
132+
mode = "true"
133+
allowImpersonation = "true"
134+
}
135+
})
136+
]
137+
138+
set_sensitive = [
139+
{
140+
name = "oauth.clientId"
141+
value = local.tailscale_oauth_client_id
142+
},
143+
{
144+
name = "oauth.clientSecret"
145+
value = local.tailscale_oauth_client_secret
146+
},
147+
]
148+
149+
depends_on = [
150+
module.eks,
151+
]
152+
}
153+
154+
#
155+
# https://tailscale.com/kb/1437/kubernetes-operator-api-server-proxy#configuring-a-high-availability-api-server-proxy
156+
#
157+
# Remove or comment out the `null_resource` provisioners that deploy `tailscale-api-server-ha-proxy.yaml` for the
158+
# high availability API server proxy to run from other platforms.
159+
#
160+
resource "null_resource" "kubectl_ha_proxy" {
161+
count = local.enable_ha_proxy_service ? 1 : 0
162+
163+
triggers = {
164+
region = data.aws_region.current.region
165+
cluster_arn = module.eks.cluster_arn
166+
cluster_name = module.eks.cluster_name
167+
ha_proxy_service_name = local.ha_proxy_service_name
168+
}
169+
170+
#
171+
# Create provisioners
172+
#
173+
provisioner "local-exec" {
174+
command = "aws eks update-kubeconfig --region ${self.triggers.region} --name ${self.triggers.cluster_name}"
175+
}
176+
provisioner "local-exec" {
177+
command = "HA_PROXY_SERVICE_NAME=${self.triggers.ha_proxy_service_name} envsubst < ${path.module}/tailscale-api-server-ha-proxy.yaml | kubectl apply --context=${self.triggers.cluster_arn} -f -"
178+
}
179+
180+
#
181+
# Destroy provisioners
182+
#
183+
provisioner "local-exec" {
184+
when = destroy
185+
command = "aws eks update-kubeconfig --region ${self.triggers.region} --name ${self.triggers.cluster_name}"
186+
}
187+
provisioner "local-exec" {
188+
when = destroy
189+
command = "HA_PROXY_SERVICE_NAME=${self.triggers.ha_proxy_service_name} envsubst < ${path.module}/tailscale-api-server-ha-proxy.yaml | kubectl delete --context=${self.triggers.cluster_arn} -f -"
190+
}
191+
192+
depends_on = [
193+
module.vpc, # prevent network changes before this finishes during a destroy
194+
helm_release.tailscale_operator,
195+
]
196+
}

‎terraform/aws/aws-eks-operator/outputs.tf‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
output "vpc_id" {
2+
description = "VPC ID where the EKS cluster is deployed"
3+
value = module.vpc.vpc_id
4+
}
5+
6+
output "cluster_name" {
7+
description = "EKS cluster name"
8+
value = module.eks.cluster_name
9+
}
10+
11+
output "operator_namespace" {
12+
description = "Kubernetes namespace where Tailscale operator is deployed"
13+
value = kubernetes_namespace_v1.tailscale_operator.metadata[0].name
14+
}
15+
16+
output "operator_name" {
17+
description = "Configured name of the Tailscale operator"
18+
value = helm_release.tailscale_operator.name
19+
}
20+
21+
output "ha_proxy_service_name" {
22+
description = "Configured name of the Tailscale operator HA Service Proxy"
23+
value = local.ha_proxy_service_name
24+
}
25+
26+
output "cmd_kubeconfig_aws" {
27+
description = "Command to configure kubeconfig for public access to the EKS cluster"
28+
value = "aws eks update-kubeconfig --region ${data.aws_region.current.region} --name ${module.eks.cluster_name}"
29+
}
30+
31+
output "cmd_kubeconfig_tailscale" {
32+
description = "Command to configure kubeconfig for Tailscale access to the EKS cluster"
33+
value = "tailscale configure kubeconfig ${helm_release.tailscale_operator.name}"
34+
}
35+
36+
output "cmd_kubeconfig_tailscale_ha" {
37+
description = "Command to configure kubeconfig for Tailscale access to the EKS cluster using the HA Service Proxy"
38+
value = "tailscale configure kubeconfig ${local.ha_proxy_service_name}"
39+
}

‎terraform/aws/aws-eks-operator/tailscale-api-server-ha-proxy.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# https://tailscale.com/kb/1437/kubernetes-operator-api-server-proxy#configuring-a-high-availability-api-server-proxy
2+
apiVersion: tailscale.com/v1alpha1
3+
kind: ProxyGroup
4+
metadata:
5+
name: ${HA_PROXY_SERVICE_NAME}
6+
spec:
7+
type: kube-apiserver
8+
replicas: 2
9+
tags: ["tag:k8s"]
10+
kubeAPIServer:
11+
mode: auth

‎terraform/aws/aws-eks-operator/variables.tf‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
variable "tailscale_oauth_client_id" {
2+
description = "Tailscale OAuth client ID"
3+
type = string
4+
sensitive = true
5+
6+
validation {
7+
condition = length(var.tailscale_oauth_client_id) > 0
8+
error_message = "Tailscale OAuth client ID must not be empty."
9+
}
10+
}
11+
12+
variable "tailscale_oauth_client_secret" {
13+
description = "Tailscale OAuth client secret"
14+
type = string
15+
sensitive = true
16+
17+
validation {
18+
condition = length(var.tailscale_oauth_client_secret) > 0
19+
error_message = "Tailscale OAuth client secret must not be empty."
20+
}
21+
}

0 commit comments

Comments
 (0)