Skip to content

Commit 587ee8e

Browse files
lpiwowarclaude
andcommitted
Clean up certificate handling
There were multiple issues with how the operator handled certificates: 1) The lightspeed-stack pod used REQUESTS_CA_BUNDLE and SSL_CERT_FILE environment variables, which bypassed the system-configured certificates. 2) When a user provided a custom CA certificate, it was expected under the cert.crt key in their ConfigMap. This was undocumented and the required key name was not obvious. 3) PostgresDB appeared to be configured for mTLS because ssl_ca_file was set in postgres.conf and the openshift-service-ca.crt was mounted into the PostgresDB pod. This created a false sense of mTLS being in place, but with the default pg_hba.conf, client certificate verification [1] is not enabled. Neither OGX [3][4] nor Lightspeed Stack [5] supports providing client certificates to PostgresDB. 4) PostgresDB SSL connection settings were configured for OGX even though they have no effect. OGX does not support configuring the SSL mode for its PostgresDB connection [3][4], so the PostgresDB certificate verification cannot be strictly enforced on the OGX side (the default is "prefer" [2], which does not enforce certificate verification and can fall back to unencrypted communication). OGX uses a non-strict config mode, so unrecognized options are silently ignored. 5) The operator did not watch for changes to ConfigMaps. When the content of the CA bundle ConfigMap was updated, the operator did not automatically reconcile. 6) Not a bug strictly speaking, but Lightspeed Stack used ssl_mode "require" when it could have used "verify-full", which checks both that the certificate is signed by a trusted CA and that the server hostname matches the CN field in the certificate. This commit simplifies certificate handling with the following changes: - Introduce a single CA bundle ConfigMap (openstack-lightspeed-ca-bundle) containing the system CAs, user-provided CA certificates from the OpenStackLightspeed CRD, kube-root-ca.crt, and openshift-service-ca.crt. This bundle is mounted into all containers in the lightspeed-stack-deployment pod, eliminating the need for REQUESTS_CA_BUNDLE and SSL_CERT_FILE. - When a user specifies a ConfigMap with custom CA certificates, iterate over all keys, validate that each holds a valid certificate, and append it to the CA bundle (resolves #2). - Stop mounting openshift-service-ca into the Postgres pod and remove ssl_ca_file from postgres.conf. These gave a false sense of client certificate validation; actually enforcing it requires configuring pg_hba.conf [1], and neither OGX [3][4] nor Lightspeed Stack [5] currently supports providing client certificates. - Remove ssl_mode, ca_cert_path, and gss_encmode from storage.backends.postgres_backend in ogx_config.yaml. These options are not supported by OGX [3][4] and gave a false sense of SSL being configured. - Add a Watch() on ConfigMaps to the reconciler so that whenever a user updates the CA bundle ConfigMap, the reconcile loop runs automatically. - Configure Lightspeed Stack with ssl_mode "verify-full" for its PostgreSQL connection, ensuring both CA trust and hostname verification. [1] https://www.postgresql.org/docs/current/ssl-tcp.html#SSL-CLIENT-CERTIFICATES [2] https://magicstack.github.io/asyncpg/current/api/index.html#asyncpg.connection.connect [3] https://github.com/ogx-ai/ogx/blob/34d7901/src/ogx/core/storage/datatypes.py#L200 [4] https://github.com/ogx-ai/ogx/blob/34d7901/src/ogx/core/storage/sqlalchemy_sqlstore.py#L125 [5] https://github.com/lightspeed-core/lightspeed-stack/blob/7503ebd/src/models/config.py#L181 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent dbe8c44 commit 587ee8e

20 files changed

Lines changed: 1169 additions & 257 deletions

internal/controller/assets/postgres.conf

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,8 @@ huge_pages = off
22
ssl = on
33
ssl_cert_file = '/etc/certs/tls.crt'
44
ssl_key_file = '/etc/certs/tls.key'
5-
ssl_ca_file = '/etc/certs/cm-olspostgresca/service-ca.crt'
5+
# mTLS is not supported by lightspeed-stack or OGX (llama-stack) for the
6+
# database connection. Neither application supports presenting client certificates
7+
# (sslcert/sslkey) to PostgreSQL, so ssl_ca_file has no effect (even when pg_hba.conf
8+
# is correctly configured for mTLS)
9+
# ssl_ca_file = '<none>'

internal/controller/ca_bundle.go

Lines changed: 190 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,190 @@
1+
/*
2+
Copyright 2026.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
package controller
18+
19+
import (
20+
"bytes"
21+
"context"
22+
"crypto/sha256"
23+
"crypto/x509"
24+
"encoding/pem"
25+
"fmt"
26+
"os"
27+
"time"
28+
29+
common_helper "github.com/openstack-k8s-operators/lib-common/modules/common/helper"
30+
apiv1beta1 "github.com/openstack-lightspeed/operator/api/v1beta1"
31+
corev1 "k8s.io/api/core/v1"
32+
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
33+
"sigs.k8s.io/controller-runtime/pkg/client"
34+
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
35+
)
36+
37+
type caCert struct {
38+
hash [sha256.Size]byte
39+
cert *x509.Certificate
40+
expire time.Time
41+
}
42+
43+
type caBundle struct {
44+
certs []caCert
45+
}
46+
47+
// getOperatorCABundle reads the system CA bundle from the operator pod's filesystem.
48+
var getOperatorCABundle = func() ([]byte, error) {
49+
contents, err := os.ReadFile(SystemTLSCABundlePath)
50+
if err != nil {
51+
return nil, fmt.Errorf("failed to read system CA bundle: %w", err)
52+
}
53+
return contents, nil
54+
}
55+
56+
// getCertsFromPEM parses PEM data and adds valid certificates to the bundle.
57+
// Rejects non-CERTIFICATE blocks and invalid X.509 data. Skips expired certs.
58+
// Deduplicates by SHA256 hash of the raw DER bytes.
59+
func (cab *caBundle) getCertsFromPEM(pemData []byte) error {
60+
if pemData == nil {
61+
return fmt.Errorf("certificate data is nil")
62+
}
63+
64+
rest := pemData
65+
for {
66+
var block *pem.Block
67+
block, rest = pem.Decode(rest)
68+
if block == nil {
69+
break
70+
}
71+
72+
if block.Type != "CERTIFICATE" {
73+
return fmt.Errorf("invalid PEM block type %q: only CERTIFICATE blocks are permitted", block.Type)
74+
}
75+
76+
certificate, err := x509.ParseCertificate(block.Bytes)
77+
if err != nil {
78+
return fmt.Errorf("invalid certificate: %w", err)
79+
}
80+
81+
if time.Now().After(certificate.NotAfter) {
82+
continue
83+
}
84+
85+
blockHash := sha256.Sum256(block.Bytes)
86+
isDuplicate := false
87+
for _, existing := range cab.certs {
88+
if existing.hash == blockHash {
89+
isDuplicate = true
90+
break
91+
}
92+
}
93+
if !isDuplicate {
94+
cab.certs = append(cab.certs, caCert{
95+
hash: blockHash,
96+
cert: certificate,
97+
expire: certificate.NotAfter,
98+
})
99+
}
100+
}
101+
102+
if len(bytes.TrimSpace(rest)) > 0 {
103+
return fmt.Errorf("trailing non-PEM data (%d bytes)", len(bytes.TrimSpace(rest)))
104+
}
105+
106+
return nil
107+
}
108+
109+
// encodePEM encodes all certificates in the bundle back to PEM format.
110+
func (cab *caBundle) encodePEM() []byte {
111+
var result []byte
112+
for _, c := range cab.certs {
113+
block := &pem.Block{
114+
Type: "CERTIFICATE",
115+
Bytes: c.cert.Raw,
116+
}
117+
result = append(result, pem.EncodeToMemory(block)...)
118+
}
119+
return result
120+
}
121+
122+
// mergeCertsFromConfigMap reads the Data section of the given ConfigMap
123+
// and adds any valid certificate entries to the bundle.
124+
func (cab *caBundle) mergeCertsFromConfigMap(h *common_helper.Helper, ctx context.Context, cmName string) error {
125+
cm := &corev1.ConfigMap{}
126+
if err := h.GetClient().Get(ctx, client.ObjectKey{
127+
Name: cmName,
128+
Namespace: h.GetBeforeObject().GetNamespace(),
129+
}, cm); err != nil {
130+
return err
131+
}
132+
for key, certData := range cm.Data {
133+
if err := cab.getCertsFromPEM([]byte(certData)); err != nil {
134+
return fmt.Errorf("%w: key %q in ConfigMap %q: %v", ErrParseUserCA, key, cmName, err)
135+
}
136+
}
137+
return nil
138+
}
139+
140+
// reconcileCABundleConfigMap builds a CA bundle containing the operator's
141+
// system CA certificates, a user-provided CA ConfigMap (if specified), as well as
142+
// the "kube-root-ca.crt" and "openshift-service-ca.crt" ConfigMaps. It then creates
143+
// or updates the managed ConfigMap, which is mounted into application pods.
144+
func reconcileCABundleConfigMap(h *common_helper.Helper, ctx context.Context, instance *apiv1beta1.OpenStackLightspeed) error {
145+
logger := h.GetLogger()
146+
bundle := &caBundle{}
147+
148+
systemCAs, err := getOperatorCABundle()
149+
if err != nil {
150+
return fmt.Errorf("%w: %v", ErrReadSystemCABundle, err)
151+
}
152+
153+
if err := bundle.getCertsFromPEM(systemCAs); err != nil {
154+
return fmt.Errorf("%w: %v", ErrParseSystemCABundle, err)
155+
}
156+
157+
certsCMs := []string{OpenShiftServiceCAConfigMap, KubeRootCAConfigMap}
158+
if instance.Spec.TLSCACertBundle != "" {
159+
certsCMs = append(certsCMs, instance.Spec.TLSCACertBundle)
160+
}
161+
162+
for _, certCM := range certsCMs {
163+
if err := bundle.mergeCertsFromConfigMap(h, ctx, certCM); err != nil {
164+
return fmt.Errorf("%w %q: %v", ErrGetCAConfigMap, certCM, err)
165+
}
166+
logger.Info("CA certificates merged", "configmap", certCM)
167+
}
168+
169+
bundlePEM := bundle.encodePEM()
170+
171+
cm := &corev1.ConfigMap{
172+
ObjectMeta: metav1.ObjectMeta{
173+
Name: CABundleConfigMapName,
174+
Namespace: h.GetBeforeObject().GetNamespace(),
175+
},
176+
}
177+
178+
result, err := controllerutil.CreateOrPatch(ctx, h.GetClient(), cm, func() error {
179+
cm.Data = map[string]string{
180+
CABundleKey: string(bundlePEM),
181+
}
182+
return controllerutil.SetControllerReference(h.GetBeforeObject(), cm, h.GetScheme())
183+
})
184+
if err != nil {
185+
return fmt.Errorf("%w: %v", ErrCreateCABundle, err)
186+
}
187+
188+
logger.Info("CA bundle ConfigMap reconciled", "name", cm.Name, "result", result, "certCount", len(bundle.certs))
189+
return nil
190+
}

0 commit comments

Comments
 (0)