Skip to content

Commit 4b670e8

Browse files
openshift-merge-bot[bot]openshift-merge-bot[bot]
authored
Merge pull request #12 from fonta-rh/ocpedge-1966-mid-PR
Ocpedge-1966 - Instance and Cluster Lifecycle Management
2 parents 0a551a8 + e7ccea1 commit 4b670e8

80 files changed

Lines changed: 2570 additions & 177 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
deploy/openshift-clusters/roles/install-dev/files/config_arbiter.sh
2+
deploy/openshift-clusters/roles/install-dev/files/config_fencing.sh
3+
deploy/openshift-clusters/inventory-backup/

‎README.md‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,14 @@
11
# TWO-NODE TOOLBOX
22
## Introduction
3-
This repository contains tools to deploy testing or development two-node Openshift clusters.
3+
This repository contains tools to deploy testing or development two-node Openshift clusters. Most lifecycle operations can be performed directly from the [deploy](deploy/) folder using `make`. Its README file and `make help` will show you the available options.
44

55
## Creating a dev environment
6-
These tools expect a valid RHEL hypervisor to host all necessary VMs to create and set up the cluster.
7-
If you don't already have one available, the [dev-env-aws-hypervisor](deploy/dev-env-aws-hypervisor/README.md) folder provides instructions and tools to automatically create this base host in AWS.
6+
The `openshift-cluster` tools expect a valid RHEL hypervisor to host all necessary VMs to create and set up the cluster.
7+
If you don't already have one available, the [aws-hypervisor](deploy/aws-hypervisor/README.md) folder provides instructions and tools to automatically create this base host in AWS.
88

9-
Once the hypervisor is setup, you can use the [deployment tools](deploy/ipi-baremetalds-virt/README.md) to create specific OpenShift clusters within that newly provisioned host, or on your own one.
9+
Once the hypervisor is setup, you can use the [deployment tools](deploy/openshift-clusters/README.md) to create specific OpenShift clusters within that newly provisioned host, or on your own one. For AWS-hosted environments, quick deployment options are available via `make fencing-ipi` and `make arbiter-ipi` commands directly from the deploy folder.
10+
11+
Provided you have followed the instructions on both of these READMEs for unattended installation, you can create the instance and install OpenShift on it with one command. For example `make deploy arbiter-ipi` will create an Arbiter topology OpenShift cluster using the IPI method.
1012

1113
## Two-node available topologies
1214
[Two-Node with Arbiter](docs/arbiter/README.md) and [Two-Node with Fencing](docs/fencing/README.md) topologies are available. You can read more on them in the [docs](docs) folder.

‎deploy/Makefile‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
create:
2+
@./aws-hypervisor/scripts/create.sh
3+
4+
deploy:
5+
@$(MAKE) create init inventory
6+
7+
init:
8+
@./aws-hypervisor/scripts/init.sh
9+
10+
destroy:
11+
@./aws-hypervisor/scripts/destroy.sh
12+
13+
start:
14+
@./aws-hypervisor/scripts/start.sh
15+
16+
stop:
17+
@./aws-hypervisor/scripts/stop.sh
18+
19+
force-stop:
20+
@./aws-hypervisor/scripts/force-stop.sh
21+
22+
redeploy-cluster:
23+
@./openshift-clusters/scripts/redeploy-cluster.sh
24+
25+
shutdown-cluster:
26+
@./openshift-clusters/scripts/shutdown-cluster.sh
27+
28+
startup-cluster:
29+
@./openshift-clusters/scripts/startup-cluster.sh
30+
31+
clean:
32+
@./openshift-clusters/scripts/clean.sh
33+
34+
full-clean:
35+
@./openshift-clusters/scripts/full-clean.sh
36+
37+
ssh:
38+
@./aws-hypervisor/scripts/ssh.sh
39+
40+
info:
41+
@./aws-hypervisor/scripts/print_instance_data.sh
42+
43+
inventory:
44+
@./aws-hypervisor/scripts/inventory.sh
45+
46+
fencing-ipi:
47+
@./openshift-clusters/scripts/deploy-fencing-ipi.sh
48+
49+
arbiter-ipi:
50+
@./openshift-clusters/scripts/deploy-arbiter-ipi.sh
51+
52+
help:
53+
@echo "Available commands:"
54+
@echo ""
55+
@echo "Instance Lifecycle Management:"
56+
@echo " deploy - Create, initialize, and update inventory for new EC2 instance"
57+
@echo " create - Create new EC2 instance"
58+
@echo " init - Initialize deployed instance"
59+
@echo " start - Start stopped EC2 instance"
60+
@echo " stop - Stop running EC2 instance gracefully (interactive cluster handling)"
61+
@echo " force-stop - Force stop EC2 instance immediately (no cluster checks)"
62+
@echo " destroy - Completely destroy EC2 instance and all resources"
63+
@echo ""
64+
@echo "Instance Utils:"
65+
@echo " ssh - SSH into the EC2 instance"
66+
@echo " info - Display instance information"
67+
@echo " inventory - Update inventory.ini with current instance IP"
68+
@echo ""
69+
@echo "OpenShift Cluster Management:"
70+
@echo " fencing-ipi - Deploy fencing IPI cluster (non-interactive)"
71+
@echo " arbiter-ipi - Deploy arbiter IPI cluster (non-interactive)"
72+
@echo " redeploy-cluster - Redeploy OpenShift cluster using dev-scripts make redeploy"
73+
@echo " shutdown-cluster - Shutdown OpenShift cluster VMs in orderly fashion"
74+
@echo " startup-cluster - Start up OpenShift cluster VMs and proxy container"
75+
@echo " clean - Clean OpenShift cluster using dev-scripts clean target"
76+
@echo " full-clean - Fully clean instance cache and OpenShift cluster using dev-scripts realclean target"
77+

‎deploy/README.md‎

Lines changed: 293 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,293 @@
1+
# Deploy
2+
3+
This directory contains deployment tools and scripts for setting up EC2 instances and OpenShift clusters for development purposes.
4+
5+
## Prerequisites
6+
7+
### AWS CLI
8+
If you are using the EC2 hypervisor option you will need to have the AWS CLI configured and the `AWS_PROFILE` environment variable set.
9+
10+
For getting and configuring the CLI: https://docs.aws.amazon.com/cli/
11+
12+
You can check if you have the AWS CLI properly configured by running:
13+
14+
```bash
15+
$ aws configure list
16+
Name Value Type Location
17+
---- ----- ---- --------
18+
profile openshift-dev env ['AWS_PROFILE', 'AWS_DEFAULT_PROFILE']
19+
access_key ****************4SU3 shared-credentials-file
20+
secret_key ****************z0DF shared-credentials-file
21+
region us-east-2 config-file ~/.aws/config
22+
```
23+
24+
### Dependencies
25+
The following programs must be present in your local environment:
26+
- make
27+
- aws
28+
- jq
29+
- rsync
30+
- golang
31+
32+
Also:
33+
- .ssh/config file must exist
34+
35+
#### Extra dependencies
36+
For automatic Redfish Pacemaker configuration on 4.19, you also need:
37+
- Python3 kubernetes library (https://pypi.org/project/kubernetes/)
38+
39+
Additionally, if you're using Mac OS, you might not have `timeout`, so you might also need to install coreutils, for example via brew:
40+
`brew install coreutils`
41+
42+
## Configuration
43+
44+
Before deployment, configure your environment by setting up the `aws-hypervisor/instance.env` file. Copy `aws-hypervisor/instance.env.template` to `aws-hypervisor/instance.env` and set all variables to valid values for your user.
45+
46+
## Available Commands
47+
48+
To see all available commands, run:
49+
```bash
50+
$ make help
51+
```
52+
53+
### Quick Start
54+
```bash
55+
# Create, initialize, and update inventory for a new EC2 instance
56+
$ make deploy
57+
```
58+
59+
This will create the instance, initialize it, and update the inventory in one command, placing you in a login shell for the EC2 instance.
60+
61+
### Recommended Instance Reuse Workflow
62+
63+
For quickly reusing an existing instance with a fresh cluster deployment:
64+
65+
```bash
66+
# Force stop the instance (bypasses cluster checks)
67+
$ make force-stop
68+
69+
# Start the instance
70+
$ make start
71+
72+
# Deploy a fresh cluster
73+
$ make redeploy-cluster
74+
```
75+
76+
This sequence is the fastest way to reset and reuse an instance when cluster preservation is not needed.
77+
78+
### Basic Instance Operations
79+
```bash
80+
# Create new EC2 instance
81+
$ make create
82+
83+
# Initialize deployed instance
84+
$ make init
85+
86+
# Update inventory.ini with current instance IP
87+
$ make inventory
88+
89+
# SSH into the EC2 instance
90+
$ make ssh
91+
92+
# Get instance info
93+
$ make info
94+
95+
# Start a stopped instance
96+
$ make start
97+
98+
# Stop a running instance (with cluster management options)
99+
$ make stop
100+
101+
# Force stop instance immediately (no cluster checks)
102+
$ make force-stop
103+
104+
# Completely destroy the instance
105+
$ make destroy
106+
```
107+
108+
### OpenShift Cluster Management
109+
110+
When running OpenShift clusters on the instance (using dev-scripts), you have several options for managing cluster lifecycle:
111+
112+
**Quick deployment commands:**
113+
- `make fencing-ipi` and `make arbiter-ipi` provide non-interactive deployment for specific topologies
114+
- These commands automatically call the underlying setup.yml playbook with the appropriate configuration
115+
- Useful for automation and when you know exactly which topology you want to deploy
116+
117+
#### Option 1: Force Stop and Redeploy (Recommended for instance reuse)
118+
```bash
119+
# Force stop the instance immediately (no prompts, cluster will be lost)
120+
$ make force-stop
121+
122+
# Start the instance
123+
$ make start
124+
125+
# Redeploy the cluster from scratch
126+
$ make redeploy-cluster
127+
```
128+
129+
**This is the recommended workflow for quickly reusing an instance when you don't need to preserve cluster state.**
130+
131+
Alternative forcible stop method:
132+
```bash
133+
# Using interactive stop with forcible option
134+
$ make stop
135+
# Choose option 1 when prompted for forcible stop
136+
```
137+
138+
#### Option 2: Redeploy Cluster (Clean and Rebuild)
139+
```bash
140+
# Redeploy the cluster (clean existing and rebuild)
141+
$ make redeploy-cluster
142+
```
143+
144+
This option:
145+
- Automatically cleans up the existing cluster
146+
- Supports interactive mode selection (arbiter or fencing)
147+
- Intelligently detects cluster topology changes
148+
- For same topology: Uses make redeploy (fast, preserves cached data)
149+
- For topology changes: Uses make realclean + full installation (slower but clean)
150+
- Integrates with Ansible playbooks for orchestration
151+
152+
**When to use redeploy:**
153+
- When you want to refresh the cluster with the latest changes
154+
- When the cluster is in an inconsistent state
155+
- For testing deployment changes
156+
- When switching between cluster modes (arbiter ↔ fencing)
157+
158+
#### Option 3: Delete Cluster and Clean Server
159+
```bash
160+
# Delete the cluster and clean the server
161+
$ make clean
162+
163+
# Stop the instance
164+
$ make stop
165+
166+
# When restarted, you'll need to redeploy the cluster from scratch
167+
$ make start
168+
# Quick deployment over clean server
169+
$ make fencing-ipi # Deploy fencing topology
170+
$ make arbiter-ipi # Deploy arbiter topology
171+
```
172+
173+
#### Option 4: Graceful Cluster Shutdown/Startup (Not recommended due to speed and consistency)
174+
```bash
175+
# Gracefully shutdown the cluster VMs before stopping the instance
176+
$ make shutdown-cluster
177+
178+
# Stop the instance (cluster VMs are preserved in shutdown state)
179+
$ make stop
180+
181+
# Start the instance again
182+
$ make start
183+
184+
# Start up the cluster VMs and proxy container
185+
$ make startup-cluster
186+
```
187+
188+
## Cluster Management Details
189+
190+
**Important: "Clean" operations delete the cluster completely. All cluster data, configurations, and workloads will be permanently lost.**
191+
192+
### Clean Options
193+
```bash
194+
# Standard clean: Remove cluster while preserving cached data for faster redeployment
195+
$ make clean
196+
197+
# Full clean: Complete cleanup including all cached data (slower but thorough)
198+
$ make full-clean
199+
```
200+
201+
**When to Use:**
202+
- **`make clean`**: Standard cluster cleanup while preserving cached data for faster subsequent deployments
203+
- **`make full-clean`**: Complete cleanup when you want to start completely fresh or troubleshoot deployment issues
204+
205+
**When to Use Each Method:**
206+
- **Redeploy**: For changing configurations, updating cluster deployment, switching cluster modes
207+
- **Delete and Clean**: For planned maintenance, manual control over cleanup
208+
- **Force Stop**: For instance reuse with cluster reinstallation, when cluster is corrupted, or when cluster preservation is not needed
209+
- **Shutdown/Startup**: For cluster state preservation
210+
211+
## Interactive Stop Script
212+
213+
When running `make stop` on an instance with a running OpenShift cluster, you'll be presented with options:
214+
215+
1. **Shutdown the cluster VMs**: Runs `make shutdown-cluster` first (recommended)
216+
2. **Delete cluster and clean server**: Runs Ansible cleanup playbook
217+
3. **Continue with forcible stop**: Stops instance immediately (cluster lost)
218+
219+
The script automatically detects:
220+
- OpenShift dev-scripts installations
221+
- Running cluster VMs
222+
- Cluster state and provides appropriate options
223+
224+
## Instance Recovery Options
225+
226+
After restarting an instance with `make start`, you'll see guidance for:
227+
228+
**If you previously shutdown your cluster:**
229+
```bash
230+
# Start up the existing cluster
231+
$ make startup-cluster
232+
```
233+
234+
**If you need to create or redeploy a cluster:**
235+
```bash
236+
# Option 1: Automated redeploy with mode selection
237+
$ make redeploy-cluster
238+
239+
# Option 2: Manual clean and setup approach
240+
$ make clean
241+
$ make arbiter-ipi #(for example)
242+
```
243+
244+
## Troubleshooting Cluster Management
245+
246+
If cluster startup fails:
247+
```bash
248+
# Check cluster status manually
249+
$ make ssh
250+
$ cd ~/openshift-metal3/dev-scripts
251+
$ oc --kubeconfig=ocp/<cluster-name>/auth/kubeconfig get nodes
252+
253+
# If cluster is unrecoverable, clean and redeploy
254+
$ make redeploy-cluster
255+
```
256+
257+
If VMs don't start properly:
258+
```bash
259+
# Check VM states manually
260+
$ make ssh
261+
$ sudo virsh list --all
262+
$ sudo virsh domstate <vm-name>
263+
264+
# Manually start VMs if needed
265+
$ sudo virsh start <vm-name>
266+
```
267+
268+
If proxy container issues occur:
269+
```bash
270+
# Check proxy container status
271+
$ make ssh
272+
$ podman ps --filter name=external-squid
273+
274+
# Restart proxy if needed
275+
$ podman restart external-squid
276+
```
277+
278+
## Advanced Features
279+
280+
**Cluster State Tracking:**
281+
- State saved in `aws-hypervisor/instance-data/cluster-vm-state.json`
282+
- Tracks deployment mode (arbiter/fencing)
283+
- Detects configuration changes for intelligent cleanup
284+
285+
**VM Infrastructure Management:**
286+
- Automatic detection of VM configuration changes
287+
- Safe cleanup when switching between cluster types
288+
- Preservation of VM infrastructure when possible
289+
290+
**Proxy Container Management:**
291+
- Automatic proxy container lifecycle management
292+
- Integration with cluster startup/shutdown workflows
293+
- Status checking and recovery capabilities

deploy/dev-env-aws-hypervisor/.gitignore renamed to deploy/aws-hypervisor/.gitignore

File renamed without changes.

0 commit comments

Comments
 (0)