Skip to content

Commit 47d8f1c

Browse files
nedoshinedoshi
committed
Add VPCE IP discovery, VPCE SG step, and a short security disclaimer
Signed-off-by: Nerav Doshi <103272684+nedoshi@users.noreply.github.com>
1 parent da4a0da commit 47d8f1c

1 file changed

Lines changed: 148 additions & 85 deletions

File tree

‎content/rosa/hcp-public-nlb/_index.md‎

Lines changed: 148 additions & 85 deletions
Original file line numberDiff line numberDiff line change
@@ -9,11 +9,13 @@ validated_version: "4.20"
99
---
1010
## Overview
1111

12-
This document provides guidance on using a public AWS Network Load Balancer (NLB) to connect to a private Red Hat OpenShift on AWS (ROSA) Hosted Control Plane (HCP) cluster. When the cluster is private and lacks direct public IP access, the NLB facilitates secure and reliable routing of traffic from public sources to the private cluster by exposing a stable endpoint while maintaining network isolation. This setup ensures that the private cluster can effectively handle external traffic, such as requests from APIs or services, without directly exposing sensitive internal infrastructure to the Internet.
12+
This guide describes how to use an **internet-facing** AWS Network Load Balancer (NLB) to reach the **Kubernetes API** of a **private** Red Hat OpenShift Service on AWS (ROSA) Hosted Control Plane (HCP) cluster. The NLB terminates TLS on a custom domain and forwards traffic to the private IP addresses of the cluster API VPC endpoint network interfaces.
13+
14+
This pattern is different from [Securely exposing an application on a private ROSA cluster with a Network Load Balancer](/experts/rosa/hcp-private-nlb/), which adds a second ingress controller for **application** traffic in a peered public VPC. This guide targets **cluster API** access only.
1315

1416
The end-to-end traffic flow is:
1517

16-
``` bash
18+
```bash
1719
Client (Internet)
1820
→ DNS (api.example.com)
1921
→ Internet-facing NLB (TLS:443, ACM certificate for api.example.com)
@@ -22,138 +24,190 @@ Client (Internet)
2224
→ Cluster Kubernetes API
2325
```
2426

27+
{{% alert state="warning" %}}
28+
Publishing the cluster API on an internet-facing load balancer increases exposure and is not the default ROSA HCP design. Restrict source IPs on the NLB security group, use TLS with a valid certificate, monitor access, and prefer private connectivity (VPN, Direct Connect, or AWS Client VPN) when possible. See [Configuring ROSA with HCP Private Cluster API Access](/experts/rosa/hcp-private-api-access/) for PrivateLink security group patterns.
29+
{{% /alert %}}
30+
2531
## Pre-requisites
2632

27-
1. A private ROSA HCP cluster already running (4.20+) (see the [Deploying ROSA HCP documentation](https://docs.aws.amazon.com/rosa/latest/userguide/getting-started-hcp.html)).
33+
1. A private ROSA HCP cluster (4.20+) with external authentication enabled (see the [Deploying ROSA HCP documentation](https://docs.aws.amazon.com/rosa/latest/userguide/getting-started-hcp.html) and [Configuring Microsoft Entra ID as an external authentication provider](/experts/rosa/entra-external-auth/)).
34+
35+
1. [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html), [OpenShift CLI (`oc`)](https://mirror.openshift.com/pub/openshift-v4/clients/ocp/), [jq](https://stedolan.github.io/jq/download/), and the [`oidc-login`](https://github.com/int128/kubelogin) plugin for `oc`.
36+
37+
1. Microsoft Entra ID application credentials used in the Entra ID guide (`TENANT_ID`, `CLIENT_ID`, `CLIENT_SECRET`).
38+
39+
1. (Optional) A jump host or other VPC connectivity if you need to reach private resources while building the environment. See [Launch a jump host](/experts/rosa/hcp-private-nlb/rosa-private-nlb-jumphost/) if required.
40+
41+
1. A public domain you control (for example `example.com`) with a Route 53 public hosted zone in the same AWS account.
42+
43+
## Identify the cluster API VPC endpoint
44+
45+
Before creating the target group, collect the VPC endpoint ID, VPC ID, endpoint security group, and the **private IP addresses** that the NLB target group must use.
2846

29-
2. In this example, we will use Entra ID as the external authentication for ROSA HCP cluster (see [Configuring Microsoft Entra ID as an external authentication provider](/experts/rosa/entra-external-auth/))
47+
1. Set your cluster name:
3048

49+
```bash
50+
export CLUSTER_NAME=<cluster_name>
51+
```
3152

32-
3. (Optional) Launch a Jump Host EC2 instance in Public NLB VPC
53+
1. Resolve the cluster API VPC endpoint and VPC:
3354

34-
This guide requires connectivity to the cluster. Since we are using a private cluster you must ensure that your workstation is connected to the AWS VPC hosting the ROSA cluster. If you already have this connectivity through a VPN, Direct Connect or other method you can skip this section. If you do need to establish connectivity to the cluster [these instructions](/experts/rosa/hcp-private-nlb/rosa-private-nlb-jumphost/) will guide you through creating a jump host and connect to the ROSA HCP cluster.
55+
```bash
56+
read -r VPCE_ID VPC_ID <<< $(aws ec2 describe-vpc-endpoints \
57+
--filters "Name=tag:api.openshift.com/id,Values=$(rosa describe cluster -c ${CLUSTER_NAME} -o yaml | grep '^id: ' | cut -d' ' -f2)" \
58+
--query 'VpcEndpoints[].[VpcEndpointId,VpcId]' --output text)
59+
echo "VPCE_ID=${VPCE_ID} VPC_ID=${VPC_ID}"
60+
```
3561

36-
4. A public domain you control (for example example.com) with a Route 53 public hosted zone in the same AWS account
62+
1. List the VPC endpoint network interface IP addresses for the target group:
3763

38-
## Create a security group, a target group and network load balancer in your AWS account
64+
```bash
65+
VPCE_IPS=$(aws ec2 describe-network-interfaces \
66+
--filters "Name=vpc-endpoint-id,Values=${VPCE_ID}" \
67+
--query 'NetworkInterfaces[].PrivateIpAddress' \
68+
--output text)
69+
echo "Register these IPs in the target group: ${VPCE_IPS}"
70+
```
3971

40-
Once the ROSA HCP cluster is installed with external authentication using Entra ID we need to set up an additional security group to grant access outside the VPC, as well as create target group and a NLB.
72+
1. Note the security group currently attached to the VPC endpoint:
4173

42-
### Create a Security Group for NLB:
74+
```bash
75+
VPCE_SG_ID=$(aws ec2 describe-vpc-endpoints \
76+
--vpc-endpoint-ids ${VPCE_ID} \
77+
--query 'VpcEndpoints[0].Groups[0].GroupId' \
78+
--output text)
79+
echo "VPCE_SG_ID=${VPCE_SG_ID}"
80+
```
4381

44-
Navigate to the **Security Groups** section in the AWS console click on **Create security group**.
82+
Keep `VPCE_IPS` available for the target group step below.
4583

46-
- **Name tag**: Give your security group a name. Select the VPC in which your Network Load Balancer is in.**Click Create**.
47-
- **Modify Inbound rules** Select the newly created security group from the list. Navigate to the **Inbound rules** tab and click **Edit Inbound rules**. Add a new inbound rule with the following settings:
48-
- **Type**: Choose the appropriate protocol for your NLB (e.g., HTTP, HTTPS, or TCP, depending on the service you're exposing).
49-
- **Protocol**: Choose the protocol for your NLB (TCP is commonly used for NLBs).
50-
- **Port Range**: Specify the port on which your NLB is listening on (e.g., 80 for HTTP, 443 for HTTPS).
51-
- **Source**:
52-
- Choose **My IP** to allow access from your current IP address.
53-
- Alternatively, specify a custom IP range in CIDR format (e.g., `192.x.x.x/24` for a specific subnet).
84+
## Create a security group, target group, and network load balancer
5485

55-
example output from the AWS console :
86+
Once the ROSA HCP cluster is installed with external authentication using Entra ID, create an NLB security group, allow the NLB to reach the API VPC endpoint, create the target group, and create the NLB.
87+
88+
### Create a security group for the NLB
89+
90+
Navigate to the **Security Groups** section in the AWS console and click **Create security group**.
91+
92+
- **Name tag**: Choose a descriptive name (for example `rosa-hcp-api-nlb-sg`).
93+
- **VPC**: Select the VPC where the NLB and VPC endpoint targets are located (`${VPC_ID}`).
94+
- **Inbound rules**: Add a rule for **TCP** port **443** from trusted sources only (for example **My IP** or a specific CIDR). Avoid `0.0.0.0/0` unless your security review requires it.
95+
- Click **Create security group** and note the security group ID as `NLB_SG_ID`.
96+
97+
Example output from the AWS console:
5698

5799
![AWS Console Additional Security group](./images/aws-portal-sg-allow-access.png)
58100

59-
### Create a target group with VPC Endpoints as targets
101+
Or create the NLB security group with the AWS CLI:
60102

61-
Define the target group with a list of VPC endpoints IPs that an NLB can access based on configured rules and health checks; allowing for load balancing across multiple instances within the group.
103+
```bash
104+
export NLB_SG_ID=$(aws ec2 create-security-group \
105+
--description "Internet-facing NLB for ${CLUSTER_NAME} API" \
106+
--group-name "${CLUSTER_NAME}-api-nlb-sg" \
107+
--vpc-id ${VPC_ID} \
108+
--output text)
109+
aws ec2 authorize-security-group-ingress \
110+
--group-id ${NLB_SG_ID} \
111+
--protocol tcp \
112+
--port 443 \
113+
--cidr <your-trusted-cidr>
114+
```
62115

63-
Here is a step-by-step guide for creating a target group in AWS for your network load balancer:
116+
### Allow the NLB to reach the API VPC endpoint
64117

65-
1. Create a Target Group:
66-
- Navigate to the **Target Groups** section in the AWS console and click **Create target group**.
67-
- **Target type**: Select **IP addresses** if you're using IP-based routing (common for VPCEs or EC2 instances located in private subnets).
68-
- **Protocol**: Choose **TLS** to secure your communication.
69-
- **Port**: Set the **Port** to **443**, which is the standard port for secure HTTPS/TLS traffic.
70-
- **VPC**: Choose the **VPC** where your targets for ROSA HCP VPCEs are located.
118+
The API VPC endpoint security group must allow inbound **TCP 443** from the NLB security group. If you created a dedicated security group for broader API access, you can attach it to the VPC endpoint as described in [Configuring ROSA with HCP Private Cluster API Access](/experts/rosa/hcp-private-api-access/).
119+
120+
```bash
121+
aws ec2 authorize-security-group-ingress \
122+
--group-id ${VPCE_SG_ID} \
123+
--protocol tcp \
124+
--port 443 \
125+
--source-group ${NLB_SG_ID}
126+
```
71127

72-
2. Configure Health Checks:
73-
- **Health check protocol**: Set to **TCP** to verify that the backend targets are healthy and accepting connections.
74-
- **Health check port**: Set to **443**, which matches the port your targets will use for traffic.
75-
- **Health check path**: Leave this field empty or specify path (e.g., `/health`) if you want to perform HTTP/HTTPS health checks.
76-
- Optionally adjust other health check settings (e.g., threshold, interval) according to your requirements.
128+
If ingress from the NLB security group is already allowed, AWS returns an error you can ignore.
77129

78-
3. Add Targets:
79-
- **IP addresses**: Enter the **IP addresses** of the targets that should be included in this target group. You might enter the IPs of VPCEs that you want the load balancer to route traffic to private HCP cluster.
80-
- Click **Include as pending below** to add these targets to the group.
130+
### Create a target group with VPC endpoint IPs as targets
81131

82-
4. **Create the Target Group**:
83-
- After verifying all your settings, click **Create target group** to complete the setup of your target group.
132+
Define the target group with the private IP addresses of the API VPC endpoint ENIs (`${VPCE_IPS}`).
84133

85-
Once created, you can associate this target group with your NLB listener to route traffic to the ROSA HCP cluster.
134+
1. Create a target group:
135+
- Navigate to the **Target Groups** section in the AWS console and click **Create target group**.
136+
- **Target type**: Select **IP addresses**.
137+
- **Protocol**: Choose **TLS**.
138+
- **Port**: Set the **Port** to **443**.
139+
- **VPC**: Choose the **VPC** where the ROSA HCP API VPC endpoint is located (`${VPC_ID}`).
86140

87-
example output from the AWS console :
141+
1. Configure health checks:
142+
- **Health check protocol**: **TCP**
143+
- **Health check port**: **443**
144+
- **Health check path**: Leave empty for TCP health checks.
88145

89-
![AWS Console Target Group](./images/aws-portal-targetgroup.png)
146+
1. Add targets:
147+
- **IP addresses**: Enter each address from `${VPCE_IPS}` (one per Availability Zone ENI).
148+
- Click **Include as pending below** for each address.
90149

91-
### Create and configure the public NLB
150+
1. Click **Create target group**.
92151

93-
Here is a step-by-step guide for creating a Network Load Balancer (NLB) and configuring it with your domain:
152+
Example output from the AWS console:
94153

95-
1. Create the NLB:
96-
- **Scheme**: Choose **Internet-facing**. This option allows your NLB to be accessible from the internet.
97-
- **VPC**: Select the **VPC** where your ROSA HCP cluster and VPC endpoint targets are located.
98-
- **Security Groups**: Select a **security group** that allows access to your API from your IP address. This should be configured to allow inbound traffic on the port you’ll be using (usually port 443 for secure communication).
154+
![AWS Console Target Group](./images/aws-portal-targetgroup.png)
99155

100-
2. Configure Listeners and Routing:
101-
- **Protocol**: Set the **Protocol** to **TLS**, to secure communication.
102-
- **Port**: Set the **Port** to **443**, which is the standard port for HTTPS traffic.
103-
- **Target Group**: Choose the **target group** you created earlier. This will route the incoming traffic to the appropriate targets.
156+
### Create and configure the public NLB
104157

105-
3. Secure Listener Settings (Optional but recommended for HTTPS):
106-
- **Certificate (from AWS Certificate Manager (ACM))**:
107-
- Select or **create a new certificate** using AWS ACM.
108-
- This certificate should be for the **domain name** you plan to use for your externally facing API. For example, if your API will be accessible at `api.example.com`, ensure the certificate matches that domain.
158+
1. Create the NLB:
159+
- **Scheme**: Choose **Internet-facing**.
160+
- **VPC**: Select the **VPC** where your ROSA HCP cluster and VPC endpoint targets are located (`${VPC_ID}`).
161+
- **Subnets**: Select **public subnets** in at least two Availability Zones.
162+
- **Security groups**: Select **${NLB_SG_ID}** (or the NLB security group you created in the console).
109163

110-
- Leave other settings at their **default values** unless specific changes are needed.
164+
1. Configure listeners and routing:
165+
- **Protocol**: **TLS**
166+
- **Port**: **443**
167+
- **Target group**: Choose the target group you created earlier.
111168

112-
4. Create the Load Balancer:
113-
- After configuring all the settings, click **Create load balancer** to finalize the setup of your Network Load Balancer.
169+
1. Secure listener settings:
170+
- **Certificate (from AWS Certificate Manager (ACM))**: Select or create a certificate for your API domain (for example `api.example.com`).
114171

115-
5. Update Route 53:
116-
- **Create a Record**: In **Amazon Route 53**, create a new DNS **record** that points to the NLB's **DNS name**.
117-
- The record should match the **domain name** for which the ACM certificate was issued (e.g., `api.example.com`).
118-
- Use the **Alias** record type to point to the NLB. AWS provides the DNS name of your NLB, which you can directly map to the record.
172+
1. Click **Create load balancer**.
119173

120-
This configuration will ensure that traffic is securely routed from the internet to your API, leveraging the NLB to distribute traffic to your backend resources.
174+
1. Update Route 53:
175+
- Create an **Alias** record for your API domain (for example `api.example.com`) that points to the NLB DNS name.
121176

122-
example output from the AWS console :
177+
Example output from the AWS console:
123178

124179
![AWS Console NLB](./images/aws-portal-public-nlb.png)
125180

126-
### Validate the connection to the NLB
181+
## Validate the connection to the NLB
127182

128-
Validate that you can access the NLB from your machine using the API domain name. For example:
183+
Validate that you can access the NLB from your machine using the API domain name:
129184

130185
```bash
131186
export API_URL=https://api.example.com
132187
curl "${API_URL}/version"
133188
```
134189

135-
example output:
190+
Example output (values vary by cluster version):
136191

137-
```bash
138-
[ec2-user@ipaddress ~]$ curl -v https://api.example.com/version
192+
```text
139193
{
140194
"major": "1",
141-
"minor": "28",
142-
"gitVersion": "v1.28.15+ff493be",
143-
"gitCommit": "4abcdefgchijklms",
195+
"minor": "30",
196+
"gitVersion": "v1.30.7",
197+
"gitCommit": "...",
144198
"gitTreeState": "clean",
145-
"buildDate": "2024-11-23T03:11:13Z",
146-
"goVersion": "go1.20.12 X:strictfipsruntime",
199+
"buildDate": "...",
200+
"goVersion": "...",
147201
"compiler": "gc",
148202
"platform": "linux/amd64"
149203
}
150204
```
151205

152-
### Validate connection to ROSA HCP cluster's API
206+
## Validate connection to the ROSA HCP cluster API
153207

154208
If you have not already done so, set `API_URL` to the NLB endpoint (for example `https://api.example.com`).
155209

156-
Create a KUBECONFIG file with Entra ID details for a [ROSA HCP cluster with external auth enabled](/experts/rosa/entra-external-auth/). For example, create **rosa-auth.kubeconfig** with the following information:
210+
Create a kubeconfig file with Entra ID details for a [ROSA HCP cluster with external auth enabled](/experts/rosa/entra-external-auth/). For example, create **rosa-auth.kubeconfig** with the following information:
157211

158212
```bash
159213
kube_config="
@@ -191,34 +245,43 @@ users:
191245
echo "${kube_config}" > rosa-auth.kubeconfig
192246
```
193247

194-
Set the `KUBECONFIG` environment variable to the location of the `rosa-auth.kubeconfig` file. This will configure the OpenShift CLI to authenticate against the ROSA cluster with the OIDC client.
195-
248+
Set the `KUBECONFIG` environment variable to the location of the `rosa-auth.kubeconfig` file:
196249

197250
```bash
198251
export KUBECONFIG=$(pwd)/rosa-auth.kubeconfig
199252
```
200253

201-
Confirm your access to the cluster by running the following command:
254+
Confirm your access to the cluster:
202255

203256
```bash
204257
oc get nodes
205258
```
206-
example output:
207259

208-
```bash
260+
Example output:
261+
262+
```text
209263
NAME STATUS ROLES AGE VERSION
210264
ip-10-0-0-170.ec2.internal Ready worker 3h29m v1.30.7
211265
ip-10-0-1-171.ec2.internal Ready worker 3h30m v1.30.7
212266
ip-10-0-2-161.ec2.internal Ready worker 3h29m v1.30.7
213267
```
214-
To verify you are logged in as user of the group, run the following command:
268+
269+
To verify you are logged in as an Entra ID user, run:
215270

216271
```bash
217272
oc auth whoami
218273
```
219-
example output:
220-
```bash
274+
275+
Example output:
276+
277+
```text
221278
ATTRIBUTE VALUE
222279
Username XXXXXXX@redhat.com
223280
Groups [0000000000000000 system:authenticated]
224281
```
282+
283+
## Related guides
284+
285+
- [Securely exposing an application on a private ROSA cluster with a Network Load Balancer](/experts/rosa/hcp-private-nlb/) (application ingress pattern)
286+
- [Configuring ROSA with HCP Private Cluster API Access](/experts/rosa/hcp-private-api-access/) (VPC endpoint security groups)
287+
- [Configuring Microsoft Entra ID as an external authentication provider](/experts/rosa/entra-external-auth/)

0 commit comments

Comments
 (0)