feat: add skipCertCopy option for SIGHUP-based certificate rotation

This change introduces a new 'skipCertCopy' field to the CrdbCluster API
that enables zero-downtime certificate rotation using SIGHUP signals.

Changes:
- Add skipCertCopy boolean field to CrdbClusterSpec API
- Make cert-copy initContainer conditional based on skipCertCopy flag
- When skipCertCopy=true: mount certs directly from secrets for SIGHUP rotation
- When skipCertCopy=false (default): use initContainer for strict 0400 permissions
- Add comprehensive documentation in README.md
- Add example manifests demonstrating both modes
- Add detailed testing guide

When skipCertCopy is enabled:
- No initContainer is created
- Certificates are mounted directly from Kubernetes secrets
- Certificate files have 0440 permissions (group readable)
- Certificates can be reloaded via SIGHUP without pod restart

When skipCertCopy is disabled (default):
- initContainer 'db-init' copies certificates to emptyDir
- Private keys have strict 0400 permissions
- Pod restart required for certificate rotation

Trade-offs are documented in README.md to help users choose the
appropriate mode for their security and operational requirements.

Fixes: cockroachdb/helm-charts#408
Related: cockroachdb/helm-charts#423
JIRA: CRDB-42772

Author: nameisbhaskar

README.md

@@ -3,6 +3,7 @@
 The CockroachDB Kubernetes Operator deploys CockroachDB on a Kubernetes cluster. You can use the Operator to manage the configuration of a running CockroachDB cluster, including:
 
 - Authenticating certificates
+- Rotating certificates with zero downtime (SIGHUP-based rotation)
 - Configuring resource requests and limits
 - Scaling the cluster
 - Performing a rolling upgrade
@@ -72,6 +73,8 @@ On a production deployment, you should modify the `resources.requests` object in
 
 The Operator generates and approves 1 root and 1 node certificate for the cluster.
 
+For information about certificate rotation options (including zero-downtime rotation), see the [Certificate Rotation](#certificate-rotation) section.
+
 ### Apply the custom resource
 
 Apply `example.yaml`:
@@ -96,6 +99,71 @@ cockroachdb-2                         1/1     Running   0          46s
 
 Each pod should have `READY` status soon after being created.
 
+## Certificate Rotation
+
+The Operator supports two modes for handling TLS certificates:
+
+### Traditional Mode (default)
+
+By default, when `tlsEnabled: true`, the Operator uses an initContainer (`db-init`) that copies certificates from Kubernetes secrets to an emptyDir volume with strict 0400 permissions on private keys. This ensures maximum security but **requires pod restarts to rotate certificates**.
+
+Example configuration:
+```yaml
+apiVersion: crdb.cockroachlabs.com/v1alpha1
+kind: CrdbCluster
+metadata:
+  name: cockroachdb
+spec:
+  tlsEnabled: true
+  skipCertCopy: false  # or omit this field (default)
+  # ... other settings
+```
+
+### SIGHUP-Based Rotation (Zero Downtime)
+
+For zero-downtime certificate rotation, set `skipCertCopy: true`. This mounts certificates directly from Kubernetes secrets, allowing CockroachDB to reload certificates via SIGHUP signal **without pod restarts**.
+
+Example configuration:
+```yaml
+apiVersion: crdb.cockroachlabs.com/v1alpha1
+kind: CrdbCluster
+metadata:
+  name: cockroachdb
+spec:
+  tlsEnabled: true
+  skipCertCopy: true  # Enable SIGHUP-based rotation
+  # ... other settings
+```
+
+To rotate certificates with SIGHUP mode:
+
+1. Update the Kubernetes secret with new certificates:
+   ```bash
+   kubectl create secret generic <node-secret-name> \
+     --from-file=ca.crt=new-ca.crt \
+     --from-file=tls.crt=new-node.crt \
+     --from-file=tls.key=new-node.key \
+     --dry-run=client -o yaml | kubectl apply -f -
+   ```
+
+2. Wait a few seconds for Kubelet to update the mounted secret.
+
+3. Send SIGHUP to the CockroachDB process:
+   ```bash
+   kubectl exec cockroachdb-0 -- kill -SIGHUP 1
+   ```
+
+4. Repeat for all pods in the cluster.
+
+**Trade-offs:**
+
+| Mode | InitContainer | Permissions | Rotation Method | Use Case |
+|------|--------------|-------------|-----------------|----------|
+| **Traditional** (`skipCertCopy: false`) | Yes (`db-init`) | 0400 (strict) | Pod restart required | Maximum security, infrequent cert rotation |
+| **SIGHUP** (`skipCertCopy: true`) | No | 0440 (group readable) | SIGHUP signal (no restart) | Zero-downtime rotation, frequent cert updates |
+
+For a complete testing guide and examples, see [`examples/TESTING-CERT-ROTATION.md`](examples/TESTING-CERT-ROTATION.md).
+
 ## Access the SQL shell
 
 To use the CockroachDB SQL client, first launch a secure pod running the `cockroach` binary.

apis/v1alpha1/cluster_types.go

@@ -59,6 +59,14 @@ type CrdbClusterSpec struct {
 	// Default: 26257
 	// +optional
 	SQLPort *int32 `json:"sqlPort,omitempty"`
+	// (Optional) SkipCertCopy determines whether to skip the cert-copy initContainer.
+	// When true, certificates are mounted directly (allowing SIGHUP-based rotation)
+	// but may have 0440 permissions instead of 0400.
+	// When false (default), an initContainer copies certs to ensure 0400 permissions
+	// but pod restarts are required for certificate rotation.
+	// Default: false
+	// +optional
+	SkipCertCopy bool `json:"skipCertCopy,omitempty"`
 	// (Optional) TLSEnabled determines if TLS is enabled for your CockroachDB Cluster
 	// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="TLS Enabled",xDescriptors="urn:alm:descriptor:com.tectonic.ui:booleanSwitch"
 	// +optional