Skip to content

Commit 3d68f0b

Browse files
minmzzhangminmzzhang
authored
feat: support deploying ZTVP from private git repositories (#140)
* feat: support deploying ZTVP from private git repositories Add bootstrap_secrets configuration to values-secret.yaml.template with two options for private repository access: - Option A: SSH deploy key authentication - Option B: HTTPS with Personal Access Token (PAT) Add docs/private-repos.md with step-by-step deployment instructions, verification steps, and troubleshooting guidance. The common Makefile already supports TOKEN_SECRET and TOKEN_NAMESPACE; this commit provides the pattern-level configuration and documentation. Signed-off-by: Min Zhang <minzhang@redhat.com> * fix: add insecureIgnoreHostKey for SSH auth and improve troubleshooting The ArgoCD repo-server container does not have Git host SSH fingerprints in its known_hosts file, causing "knownhosts: key is unknown" errors. Add insecureIgnoreHostKey field to the SSH bootstrap_secrets template. Also document: - DISABLE_VALIDATE_ORIGIN for private repo pre-flight check - ACM temporary Degraded state during initial install (self-heals) - SSH known_hosts troubleshooting entry Signed-off-by: Min Zhang <minzhang@redhat.com> * fix: ztvp-certificates merges existing proxy CA on private repo installs When deploying from an internal Git host (e.g. gitlab.cee.redhat.com), users must add corporate CAs to proxy/cluster before install. Previously the ztvp-certificates job refused to overwrite a user-set trustedCA, leaving ACS Central unable to trust Keycloak via the ingress CA. Changes: - PHASE 8.5: include all extracted CAs (custom, additional, cluster) in the proxy CA bundle, not just ingress + service - PHASE 8.6: merge existing proxy CA ConfigMap content into ztvp-proxy-ca before taking over trustedCA management - docs/private-repos.md: document pre-install CA requirement and explain automatic merge behavior - values-secret.yaml.template: add ACM workaround bootstrap_secrets entry Signed-off-by: Min Zhang <minzhang@redhat.com> * fix: remove ACM workaround, bump ACM chart to 0.2.x The duplicate bootstrap_secrets entry targeting openshift-gitops is no longer needed. The VP operator (0.0.70+) copies credentials into vp-gitops and automatically sets global.vpArgoNamespace. The ACM chart 0.2.x reads that variable, so the private-repo policy resolves without any manual override or duplicate secret. - Remove second bootstrap_secrets entries (SSH and HTTPS workarounds) - Bump ACM chartVersion from 0.1.* to 0.2.* - Update docs/private-repos.md and values-secret.yaml.template comments Signed-off-by: Min Zhang <minzhang@redhat.com> * docs: fold origin validation skip into deploy step Move the DISABLE_VALIDATE_ORIGIN=true flag directly into the deploy command so users don't hit the git ls-remote failure before discovering the workaround in a separate section. Signed-off-by: Min Zhang <minzhang@redhat.com> * docs: address PR review - SSH origin validation and known hosts - Add DISABLE_VALIDATE_ORIGIN=true to SSH deploy step (same issue as HTTPS: Makefile git ls-remote fails against private remotes) - Document insecureIgnoreHostKey alternatives: major providers have pre-populated fingerprints; self-hosted Git requires the flag since vp-gitops namespace does not exist until install; post-install hardening via ssh-keyscan + oc patch is described Signed-off-by: Min Zhang <minzhang@redhat.com> --------- Signed-off-by: Min Zhang <minzhang@redhat.com>
1 parent b3ac165 commit 3d68f0b

4 files changed

Lines changed: 389 additions & 6 deletions

File tree

‎charts/ztvp-certificates/files/extract-certificates.sh.tpl‎

Lines changed: 42 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -342,11 +342,18 @@ fi
342342
{{- if .Values.proxyCA.enabled }}
343343
log "Creating proxy CA ConfigMap for cluster trustedCA injection"
344344

345-
# Build a bundle with ONLY the custom CAs (ingress + service).
346-
# The Cluster Network Operator automatically merges these with system CAs.
345+
# Build the proxy CA bundle with ALL extracted custom CAs so that workloads
346+
# trusting the injected bundle can reach both cluster-internal services
347+
# (via ingress/service CA) and external hosts behind corporate CAs
348+
# (via additionalCertificates / customCA).
349+
# The Cluster Network Operator merges these with the system (Mozilla) CAs.
347350
> "${TEMP_DIR}/proxy-ca-bundle.pem"
348-
for cert_file in "${TEMP_DIR}"/ingress-ca-*.crt "${TEMP_DIR}/service-ca.crt"; do
351+
for cert_file in "${TEMP_DIR}"/ingress-ca-*.crt "${TEMP_DIR}/service-ca.crt" "${TEMP_DIR}/custom-ca.crt" "${TEMP_DIR}"/*.crt; do
349352
[[ -f "$cert_file" ]] || continue
353+
# Deduplicate: skip if already appended (the *.crt glob may re-match earlier files)
354+
if grep -qF "$(head -2 "$cert_file")" "${TEMP_DIR}/proxy-ca-bundle.pem" 2>/dev/null; then
355+
continue
356+
fi
350357
cat "$cert_file" >> "${TEMP_DIR}/proxy-ca-bundle.pem"
351358
echo "" >> "${TEMP_DIR}/proxy-ca-bundle.pem"
352359
done
@@ -384,8 +391,38 @@ CURRENT_TRUSTED_CA=$(oc get proxy/cluster -o jsonpath='{.spec.trustedCA.name}' 2
384391
if [[ "$CURRENT_TRUSTED_CA" == "{{ .Values.proxyCA.configMapName }}" ]]; then
385392
log "Proxy trustedCA already set to {{ .Values.proxyCA.configMapName }}, skipping"
386393
elif [[ -n "$CURRENT_TRUSTED_CA" && "$CURRENT_TRUSTED_CA" != "{{ .Values.proxyCA.configMapName }}" ]]; then
387-
log "WARNING: Proxy trustedCA is already set to '$CURRENT_TRUSTED_CA' (not overwriting)"
388-
log "WARNING: To use ZTVP proxy CA, manually run: oc patch proxy/cluster --type=merge -p '{\"spec\":{\"trustedCA\":{\"name\":\"{{ .Values.proxyCA.configMapName }}\"}}}}'"
394+
# Merge any CAs from the existing proxy ConfigMap into our bundle so
395+
# nothing is lost when we take over trustedCA management.
396+
log "Existing proxy trustedCA ConfigMap: $CURRENT_TRUSTED_CA"
397+
EXISTING_CA_DATA=$(oc get configmap "$CURRENT_TRUSTED_CA" -n {{ .Values.global.namespace }} \
398+
-o jsonpath='{.data.ca-bundle\.crt}' 2>/dev/null || echo "")
399+
if [[ -n "$EXISTING_CA_DATA" ]]; then
400+
echo "$EXISTING_CA_DATA" > "${TEMP_DIR}/existing-proxy-ca.crt"
401+
# Append any certs not already in our bundle
402+
while IFS= read -r line; do
403+
echo "$line"
404+
done < "${TEMP_DIR}/existing-proxy-ca.crt" >> "${TEMP_DIR}/proxy-ca-bundle.pem"
405+
PROXY_BUNDLE_SIZE=$(wc -c < "${TEMP_DIR}/proxy-ca-bundle.pem" 2>/dev/null || echo 0)
406+
log "Merged existing proxy CAs into {{ .Values.proxyCA.configMapName }}"
407+
# Re-create the ConfigMap with merged content
408+
cat <<MERGEEOF | oc apply -f -
409+
apiVersion: v1
410+
kind: ConfigMap
411+
metadata:
412+
name: {{ .Values.proxyCA.configMapName }}
413+
namespace: {{ .Values.global.namespace }}
414+
labels:
415+
{{- range $key, $value := .Values.configMapLabels }}
416+
{{ $key }}: {{ $value | quote }}
417+
{{- end }}
418+
data:
419+
ca-bundle.crt: |
420+
$(cat "${TEMP_DIR}/proxy-ca-bundle.pem" | sed 's/^/ /')
421+
MERGEEOF
422+
fi
423+
log "Updating proxy/cluster trustedCA from '$CURRENT_TRUSTED_CA' to {{ .Values.proxyCA.configMapName }}"
424+
oc patch proxy/cluster --type=merge -p '{"spec":{"trustedCA":{"name":"{{ .Values.proxyCA.configMapName }}"}}}'
425+
log "OK: Proxy trustedCA configured (previous: $CURRENT_TRUSTED_CA)"
389426
else
390427
if [[ $PROXY_BUNDLE_SIZE -gt 100 ]]; then
391428
log "Setting proxy/cluster trustedCA to {{ .Values.proxyCA.configMapName }}"

‎docs/private-repos.md‎

Lines changed: 280 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,280 @@
1+
# Deploying from a Private Repository
2+
3+
This document describes how to deploy the Layered Zero Trust Validated Pattern
4+
from a private Git repository.
5+
6+
The Validated Patterns framework supports deploying from both SSH-secured and
7+
HTTPS-secured (PAT) private repositories. The mechanism works by creating an
8+
ArgoCD repository secret **before** the pattern is deployed, so that the VP
9+
operator can propagate credentials to all ArgoCD instances managed by the
10+
pattern.
11+
12+
> [!NOTE]
13+
> The upstream documentation is at
14+
> <https://validatedpatterns.io/learn/private-repos/>. This page provides
15+
> ZTVP-specific guidance that builds on the framework docs.
16+
17+
## Prerequisites
18+
19+
* An OpenShift 4.16+ cluster with `oc` CLI access
20+
* A fork or private copy of this repository
21+
* A deploy key (SSH) or Personal Access Token (HTTPS) with **read** access
22+
23+
> [!IMPORTANT]
24+
> The git remote URL in your local clone **must match** the auth type in
25+
> your `bootstrap_secrets`. The Makefile passes the remote URL to the
26+
> Pattern CR verbatim when `TOKEN_SECRET` is set:
27+
>
28+
> * SSH auth: remote must be `git@host:org/repo.git`
29+
> * HTTPS/PAT auth: remote must be `https://host/org/repo.git`
30+
>
31+
> Set with: `git remote set-url origin <matching-url>`
32+
33+
## Option A: SSH Key Authentication
34+
35+
### 1. Generate a deploy key
36+
37+
```shell
38+
ssh-keygen -t ed25519 -f ~/.ssh/ztvp-deploy-key -N ""
39+
```
40+
41+
### 2. Register the public key
42+
43+
Add `~/.ssh/ztvp-deploy-key.pub` as a **deploy key** in your Git hosting
44+
provider (GitHub Settings -> Deploy keys, GitLab Settings -> Repository ->
45+
Deploy keys, etc.).
46+
47+
### 3. Configure values-secret
48+
49+
Copy the template and uncomment the SSH `bootstrap_secrets` block:
50+
51+
```shell
52+
cp values-secret.yaml.template ~/values-secret.yaml
53+
```
54+
55+
Edit `~/values-secret.yaml` and uncomment **Option A**
56+
under the "BOOTSTRAP SECRETS" section. Update the `url` field with your
57+
repository's SSH URL:
58+
59+
```yaml
60+
bootstrap_secrets:
61+
- name: private-repo
62+
targetNamespaces:
63+
- openshift-operators
64+
labels:
65+
argocd.argoproj.io/secret-type: repository
66+
fields:
67+
- name: type
68+
value: git
69+
- name: url
70+
value: git@github.com:YOUR-ORG/layered-zero-trust.git
71+
- name: insecureIgnoreHostKey
72+
value: "true"
73+
- name: sshPrivateKey
74+
path: ~/.ssh/ztvp-deploy-key
75+
```
76+
77+
> [!NOTE]
78+
> **About `insecureIgnoreHostKey`:** ArgoCD ships with pre-populated SSH
79+
> fingerprints for github.com, gitlab.com, bitbucket.org, and
80+
> ssh.dev.azure.com in its `argocd-ssh-known-hosts-cm` ConfigMap. If your
81+
> repository is hosted on one of these providers you may omit
82+
> `insecureIgnoreHostKey` and ArgoCD will verify the host key automatically.
83+
>
84+
> For self-hosted Git servers (e.g. internal GitLab), the VP framework's
85+
> `bootstrap_secrets` mechanism does not currently support injecting entries
86+
> into `argocd-ssh-known-hosts-cm`. Since the `vp-gitops` namespace does
87+
> not exist until the VP operator creates it during install, you cannot
88+
> pre-populate known hosts beforehand. Use `insecureIgnoreHostKey: "true"`
89+
> for the initial deployment.
90+
>
91+
> As a post-install hardening step, you can add proper host verification
92+
> and remove the insecure flag:
93+
>
94+
> ```shell
95+
> ssh-keyscan gitlab.internal.example.com >> /tmp/known_hosts
96+
> oc patch cm argocd-ssh-known-hosts-cm -n vp-gitops \
97+
> --type merge -p "{\"data\":{\"ssh_known_hosts\":\"$(cat /tmp/known_hosts)\"}}"
98+
> ```
99+
100+
### 4. Deploy
101+
102+
The Makefile performs a pre-flight `git ls-remote` check against the HTTPS
103+
form of the URL. For private repos this check will fail because the local
104+
machine does not have credentials for the private remote. Pass
105+
`DISABLE_VALIDATE_ORIGIN=true` to skip it:
106+
107+
```shell
108+
./pattern.sh make DISABLE_VALIDATE_ORIGIN=true \
109+
TOKEN_SECRET=private-repo TOKEN_NAMESPACE=openshift-operators install
110+
```
111+
112+
> [!NOTE]
113+
> This is safe -- the cluster uses the `private-repo` secret (SSH key or PAT)
114+
> for actual access; the validation is only a local convenience check.
115+
116+
## Option B: HTTPS with Personal Access Token (PAT)
117+
118+
### 1. Create a PAT
119+
120+
* **GitHub:** Settings -> Developer settings -> Personal access tokens ->
121+
Fine-grained tokens. Grant **Contents: Read** on the target repository.
122+
* **GitLab:** Settings -> Access Tokens. Grant **Reporter** role with
123+
`read_repository` scope (Guest role is insufficient to clone code).
124+
125+
Store the token in a local file:
126+
127+
```shell
128+
mkdir -p ~/.config/validated-patterns
129+
echo -n "ghp_xxxxxxxxxxxxxxxxxxxx" > ~/.config/validated-patterns/git-pat
130+
chmod 600 ~/.config/validated-patterns/git-pat
131+
```
132+
133+
### 2. Configure values-secret
134+
135+
Copy the template and uncomment the HTTPS `bootstrap_secrets` block:
136+
137+
```shell
138+
cp values-secret.yaml.template ~/values-secret.yaml
139+
```
140+
141+
Edit `~/values-secret.yaml` and uncomment **Option B**
142+
under the "BOOTSTRAP SECRETS" section. Update the `url`, `username`, and
143+
`password` path:
144+
145+
```yaml
146+
bootstrap_secrets:
147+
- name: private-repo
148+
targetNamespaces:
149+
- openshift-operators
150+
labels:
151+
argocd.argoproj.io/secret-type: repository
152+
fields:
153+
- name: type
154+
value: git
155+
- name: url
156+
value: https://github.com/YOUR-ORG/layered-zero-trust.git
157+
- name: username
158+
value: YOUR-USERNAME
159+
- name: password
160+
path: ~/.config/validated-patterns/git-pat
161+
```
162+
163+
> [!NOTE]
164+
> For GitLab, the `username` must be `oauth2`, not your GitLab handle.
165+
166+
### 3. Deploy
167+
168+
For private repos the Makefile performs a pre-flight `git ls-remote` check
169+
that will fail because the local machine does not have HTTPS credentials for
170+
the private remote. Pass `DISABLE_VALIDATE_ORIGIN=true` to skip it:
171+
172+
```shell
173+
./pattern.sh make DISABLE_VALIDATE_ORIGIN=true \
174+
TOKEN_SECRET=private-repo TOKEN_NAMESPACE=openshift-operators install
175+
```
176+
177+
> [!NOTE]
178+
> This is safe -- the cluster uses the `private-repo` secret (SSH key or PAT)
179+
> for actual access; the validation is only a local convenience check.
180+
181+
## How It Works
182+
183+
1. The `bootstrap_secrets` section in `values-secret.yaml` instructs the
184+
Validated Patterns framework to create the `private-repo` Kubernetes
185+
Secret in the `openshift-operators` namespace **before** deploying the
186+
pattern.
187+
188+
2. The `argocd.argoproj.io/secret-type: repository` label tells ArgoCD to
189+
pick up the secret as a repository credential.
190+
191+
3. The `TOKEN_SECRET` and `TOKEN_NAMESPACE` Make variables set the
192+
`tokenSecret` and `tokenSecretNamespace` fields on the Pattern Custom
193+
Resource. The VP operator copies the secret as
194+
`vp-private-repo-credentials` into `vp-gitops` (its managed ArgoCD
195+
namespace).
196+
197+
4. The ACM chart (0.2.x+) `vp-private-hub-policy` copies credentials from
198+
`global.vpArgoNamespace`, which the VP operator automatically sets to
199+
`vp-gitops`. This allows the policy to find the secret the VP operator
200+
placed there without any manual override.
201+
202+
## Verifying
203+
204+
After deployment, confirm the repository secret was created:
205+
206+
```shell
207+
oc get secret private-repo -n openshift-operators \
208+
-o jsonpath='{.metadata.labels.argocd\.argoproj\.io/secret-type}'
209+
```
210+
211+
Expected output: `repository`
212+
213+
Confirm the VP operator propagated the credential to `vp-gitops`:
214+
215+
```shell
216+
oc get secret vp-private-repo-credentials -n vp-gitops \
217+
-o jsonpath='{.metadata.labels.argocd\.argoproj\.io/secret-type}'
218+
```
219+
220+
Expected output: `repository`
221+
222+
Check the Cluster ArgoCD can see the repository:
223+
224+
```shell
225+
oc get application layered-zero-trust-hub -n vp-gitops \
226+
-o jsonpath='{.status.sync.status}'
227+
```
228+
229+
Expected output: `Synced` (or `OutOfSync` if you have uncommitted changes).
230+
231+
## Troubleshooting
232+
233+
* **ACM shows Degraded (vp-private-hub-policy NonCompliant)** -- The ACM
234+
chart 0.1.x has `openshift-gitops` hardcoded in the private-repo policy
235+
template, but the VP operator (0.0.70+) places credentials in
236+
`vp-gitops`. Ensure `values-hub.yaml` uses ACM chart 0.2.x or later
237+
(`chartVersion: 0.2.*`), which reads `global.vpArgoNamespace` -- a value
238+
the VP operator sets automatically.
239+
240+
* **ArgoCD shows "repository not accessible"** -- Verify the SSH key or PAT
241+
has read access. For SSH, confirm the key has no passphrase (`ssh-keygen
242+
-y -f ~/.ssh/ztvp-deploy-key` should not prompt).
243+
244+
* **SSH: "knownhosts: key is unknown"** -- The `insecureIgnoreHostKey: "true"`
245+
field is missing from the bootstrap secret. The ArgoCD repo-server runs
246+
in a container without your Git host's fingerprint in known_hosts.
247+
248+
* **HTTPS: "x509: certificate signed by unknown authority"** -- This
249+
affects internal/self-hosted GitLab instances whose TLS certificates are
250+
signed by a corporate CA. GitHub and public GitLab (`gitlab.com`) use
251+
publicly trusted CAs and do not require this step.
252+
253+
The corporate CA must be in the cluster trust store **before** install
254+
because the VP operator needs it to clone the repository. Add the internal CA
255+
as a pre-install step:
256+
257+
```shell
258+
oc create configmap custom-ca -n openshift-config \
259+
--from-file=ca-bundle.crt=/path/to/corporate-ca-bundle.pem
260+
oc patch proxy/cluster --type=merge \
261+
-p '{"spec":{"trustedCA":{"name":"custom-ca"}}}'
262+
```
263+
264+
Wait a few minutes for operator pods to restart with the updated bundle.
265+
266+
> [!NOTE]
267+
> After the pattern deploys, the `ztvp-certificates` chart automatically
268+
> merges your `custom-ca` content into its managed `ztvp-proxy-ca`
269+
> ConfigMap and switches `proxy/cluster.spec.trustedCA` to
270+
> `ztvp-proxy-ca`. This adds the cluster ingress and service CAs so
271+
> that workloads like ACS Central can reach Keycloak without additional
272+
> manual steps. You do **not** need to manually add the ingress CA to
273+
> your `custom-ca`.
274+
275+
* **Secret not found during install** -- Ensure you ran
276+
`./pattern.sh make load-secrets` *after* the bootstrap secret was created.
277+
The `TOKEN_SECRET` and `TOKEN_NAMESPACE` values must match exactly.
278+
279+
* **GitLab HTTPS fails** -- Remember that GitLab PAT auth requires
280+
`username: oauth2`, not your GitLab user handle.

‎values-hub.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -276,7 +276,7 @@ clusterGroup:
276276
namespace: open-cluster-management
277277
project: hub
278278
chart: acm
279-
chartVersion: 0.1.*
279+
chartVersion: 0.2.*
280280
annotations:
281281
argocd.argoproj.io/sync-wave: "5"
282282
ignoreDifferences:

0 commit comments

Comments
 (0)